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ABSTEACT . , . . - 

^- ^ \ l^he purpose of this monograph is to aid practitioners 

in gaining federal Joint Dissemiiiat.ion Eeview Panel (JDEP) -approval 
of career education project^^^^ helps them understand the importance 
of designing and implementinV|^sound evaluation from the -eaxly 
stages of the program, alerts^ them to common errors or omissions that 
weaken or destroy conclusions that can be drawn from the evaluation 
results and shows how to avoid th'em, and explains administrative 
procedures involved in obtaining JDEP 'approval. Section 1, on the 
benefits of JDEP approval,, discusses structure of the National \ 
Diffusion Network (NDN) and JDEP, problems in career education 
evaluation, and OjVercoming those problem?* Section 2 contains a. 
checklist giving essential pof-gts for local career educators to 
consider in planning evaluation designs. Each item 'i"s discussed, 
Sectidfe 3 provides specific recommendations for the local career 
educator regarding the preparation and submission; of an evaluation 
report to the JDEP* In .section 4= are ccmmon questions asked by 
educators and the answers typically given, "Appendixes contain the 
JDEP submissi^dn format, three sample JCEP submissions, and a list of 
references. Each sample is the exact ten-page report submitted .and 
approved by a unanimcus or nearly unanimous vote of^the panel, 
(YIB) 
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The information reported herein was obtained pur- 
suant to Conttact No. 300-79-0549 with the Office 
of Education, U.S. Dep^artment. of Health, Education, 
and Welfare. Contractors undertaking such projects 
under Goyerninent spoi^sorsHip are encouraged to docu- 
ment xinforraation^ according to their observation and 
professional judgment. Consequently, infprmatic 
poititas of view, or opinions staped do not necessarily 
represent of ficial/of f^e of Education position r 
policy./ y 
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. IntroductjLQn? 



:^ Jean Worth- is the directpr of a 'career educatiipn. ' 

^ ptoj ect^ that - has Kad , ^eyeral • year's of implemenf^- ^ - 
tion. Slie believes- that "the proj ect is 'soiiijid", leaving , 
cbppleted exterisiye /evaluation to demonstrate ' its " ' 

• effectiveri'ess. 'With data in handV " 

she decides to try foi-.< approval by, the federal- Joint 

-!l)issei]aitiaj:i6n Review Panel (J13RP)v It isri^t th^ end 

{of the^world' to go for ^ not?^' Moreover; - 

her school priricipaL/knd distrrct superintendent 
encourage her to-m^ake thja^ffort. Jean and fie r hard- 
working staff would appreciate the^ honor of national 

jrecognition ;as ,ah ; exemplary career education activity. 
Not, only that^ it would/ also open the^-door to entry 
into the National Diffusion^ Network (NDN); Once. In 
the OT)N, others (students >nd teachers) across the 
country -could benefit .from all the work Jean and 
staff have done^ ♦ ' 

. ' • ' ■ ,■ " ' . ^ ■ ' . ., . 

Understandihg^that; before she, can submit tlje 
required 10-page summary' report on the ef f^ctivene^ss 
of her project to the JDRP, she must get the proposal 
sponsored by. a federal education program,r Jean. ta 
pen In hand. *It doesn' t take long before she /realizes 
what a chore it as to -^ull together even a first draft 
of the submission^ ; .Long before this J^n* had had 'her 
^taff score the data zcllection instruments, tabulate 
th^ data, and analyze r::iem. It had' been, kn arduous 
.task, but: tables df r^^ults are finally ready for her 
to insert into the e-':Uui^tion report. Tables, yes! 
The harder part, thc:-zZy has fallen to Jean— writing 
tha narrative that wa-_ld explain to the reader the' 
meting of »all the i^ii^lec data, not to mention the ' 
adminJ ^.trative anr -z.. rructional procedures used in 
the p: 'act . its c^■<,z::... criginr: , demographic charac-^ 
terisri £itc. 

s ■ ' * 

At long last ( ^afr^r much sweat) Jean has 'a 
rough draft of thi^^rr r:;)!-: ^eady, and she inails a 
copy of it off fi:^ : rr- state career education 
coordinator as reqiX-- i - : O.E. 's Office of Career 
^Iducaticr. CEV Jt-^^: - : e: ahds that the coordi- 
nafor-wlli :.iiv- a mer- . " ;:r staff with expertise 



^ in evaluation review^ the , draft, and that a'.writren 
summary o'f the review will eventually be mailed 
back to her. Jean is hoping that . this review will 
be brief, if not perfunctory,' so that she can get on 
with the business of submitting affinal draft to 

-the federal Office of Career Education. " She also 
understands that program officials at OCE are respon- 
sible for conducting a detailed review of the sub- 
mission before it is /approved and forwarded by them 
to tfhe JDRP. 

. It waSf in this frame of mind that Jean later' 
opened an envelope fi^m the state care.er education 
coordinator and' read the' folloying letter, which the 
coordinator Tiad forward4d to her: 



Dear Ms. Worth: . . , - • 

My staff ,q.nd L have gone over your draft 
JDRP submission, and I would like to- offer 
you the following comments'. While you have 
planned your evaluation -reasonably well, 
you have allowed inaccuracies of in-consisten- 
cies of one kind or afiothel: to be introduced 
that have clouded the results. We have * 
several areas of concern: 

lection af Evatluation. Sample : You used 
r-^tment/control i.group des:igri,^but you failed 
CO :;siiablish the coinparability of "b}ie groups. 
:ciflcally, a bias was introduced when you 
^ed volunteers for the treatment sample, ^n 
'ition, you failed . to ^^account for the greater 
iber of dropouts from the treatment group 
' compared With the cont^rol group. ' 



select ion/Devfilopmerit of Instruments : You 
■AiL c a good defense for your decision not -to 
us i tandari2:ed tests, arguing t^at available 
In:^ uments addressed your activity's objectives 
only peripherally. How^^r, the instruments you 
developed and used loc^ly were not supported by 
sat:, tdctory evidence of reliability and validity. 



^ Inferring that the Treatment Caused the 
■ Gains ; Inferring that gains made by the • . 
target population were due to the treatment, 
without ruling out the possibility of others 
reasons for the gains, is another concern. \ 
Your selection of^r-control groups did not rule 
out the possibility that intervening variable 
such as parallel programs in the school or 
, community could have influenced the results 
demonstrated by testing*. 

/' 

If ycru would ;Like to call cr write me, 
I would be happy~td /elaborate on tliese and- 
other less critical ar^eas of cor. jem in ; our 
draft submission, ' j 

Sincerely, 

* . B. Sharp 
' Evaluate r> ;on - \.ii:^ni: 



r-^fter picking herself up off the flc r, Jea.; 
: ^_ly unloaded her feelings on that st::re depar- 
r z consultant who had the gall to reejr to her 
'TLJt like that, ^^What riglit die he have to crit 
i:ir.e Jean'^s efforts when he had no firs.t-hand ir: 
:cion about her project ^at all. He was just 
lot of the taxpayer3' money. After she coolec 
:wn somewhat, Jean suppose^ that ^i^t probably wc 
:: tr worth the price of a phone call to find but : 
Lhere was any hope-^at all of correcting the prob 
noted in the letter,. . 

: t : ' 

WORTH: Hello, Dr. Sharp. This is Jean Worth 
^ and I have just, received your letter 
in which you dritiqued our draft JDRF 

submission. I'd like to kno.w more 

' ' - about what ypu think we could do to 

strengthen our evaluation data." 




SHARP: 



Jean, I'm afraid that' I can't.be' very 
encouraging. Your best bet wou^d 
have been to select the grpups ran- 
domly from'tcJtal populations, or at 
least to have established the relevant 
variables and located samples matched 
on those variables. And with your 
excessive numbe^r of dropouts ffom r le 
treatr- group, it would h^ve beer, 
bfc: ic hava limited the ana^ytsis - 
t. mosf:? wl*)'^ participated b;;r:h pre-' 
azid vo .—irement, proviaied :hat 

^ttrijx^i .cic not result in rn^^.^iematic 
bias 'iOiCwe^-u :ne treatmentVinn control 
groups 



SHARP: 



WORTH: Well, . Sr-rro, that's eaE_: ^ -aid ^ 
than .ii/:ae. " Thisre's not much .. .f I - 
can a bout tnose decisions nov... 
What a:Dc iLfe the problems you :?av ./ith 
the ir r _ 'jments .we developec 

To -£r-Tt rhe validity cvf -ou- in- 

str^jntT— , Jean, you need tc sh that 
out . ; eixperts -or studies cDnf rmed 
a - i-i specifications than r 
in^rirumienxs had to meet, anc ::..zz 
it-r.. _. vera written expressly r:: test 
for . :;aiiiineni: of the instructi -aal ' 
objeuCives of your pxoject inate-rials. 
A systematic reliability study would 
also be necessary to report, for 
example, datSu establishing' the test- ' 
retest reliability, of. the instruments. 

You're ^ery discouraging. Dr. Sharp. - 
What do you. e:xpect me to dp, sT^t 
»wall over again? » 

I'm sorry to have to be^sb disco^r- 
agirrg, Jean,' but there's no way that 
your present draft 'will be appro.ved 
by the JDRP. .You/d best go back to . . 
.the drawing .>bard ^and collect seme 



WORTH: 



SHARP: 



more data based on a Ijetter evaluation 
design. ' . 



Jean hung up the pjione and dumped backward in her 
chair. Her prpject ''funds v/bre -almost expended. There 
^ was no time or TOoney^ to -l^egin anew with a more compre- 
hensive field test/of the project. " Jean's visiori of " ' 
JDRP approval for the project and eventual funding ' 
for national dissemination had been aborted, despite 
her strong belief in the ^effectiveness of the project. 
Jean's^final disgruntledi^omments went something like 
this: "Who needs all 'this pain? I sure don't. The 
JDRP can get along just fine without pur project. • 
Though it just about killed me .writing it, the best 
place for this draft submission is. File Thirteen— * ' 
Forget it ! !" 

The purpose of this monograph is to help local 
project directors and state career education coordi- 
nators avoid the abov4 disaster by: (1) helping you, 

' the Practitioner, understand -t^he importanc^^ of design- 
ing^^ implementing a sound ev^aluatidn^ f rom the early 
stages of your program; (2) alerting you to common 

^ errors or omissions that weaken or destroy conclusions ' 
that can be 'drawn, from the evaluation results, and 
showing you how these can be aviylAei; and (3) explaining 
in step^by-step s'detaijL the administrative procedures 
involved in obtaining JDRP approval of your careet edu^^il^;' 
cation project. ' ■ * " 



Why, Go For JDRP. Approval? 



The Career Education Incentive Act , (PL95-207) ^ 
is providing approximately '2p^million , dollars per • 
year to state^educational agencies which, in turn, \^ 
award^ about 85% of that money -in grants^ to local edur 
cational agencies to enable them to implement Career ; 
education activities. Local^ school districts are 
encouraged to plan and carry out carefully designed 
evaluations of the local p^^oject * activl^es. "^Some of | 
thesfe local ^valuations may produce convincing evidenc 
pf the ef j^tiveffess of the cai*eer education activitife 





that are being conducted, providing the basis for 
th^ir consideration 6y the- Joint Dissemination Revfew 
Panel (JDRP) , Those projects that are approved as 
^exemplary by the JDRP can then be disseminated for 
replication by other ^ocal school districts. 

The Vision ' ^ 

Projects with the potential of becoming exemplary, 
such as career education projects sup|)orted with In- 
centive Act funds, may develop programs which later 
can bb'adopted or ad§p.ted by other school systems. 
An^ implied responsibility of potentially exemplary 
programs is to provide potential users with sound 
bases for deciding whether to. try them 4n their x>wn 
school systems. That is, a school administrator in 
New York shoul<t.be able to predict whether a program 
developed in California will work' in his or heir 
school on the basis of the California program's eval- 
uation results. Career education evaluations, should 
include both process and outcome data so that it is 
possible to make 5^1iable predictions from one program 
implementati^^^^. another. This includes a gdod 
description , of the treatment (s) applied thus facili- 
tating the process of transporting successful compo- 
neifts ^rom one site to other sites. 




The vision of U.S.O.E.'s Office of £are&r Education 
the agency, responsible for administering the Incentive 
Act fupds, was presented by the OCE Director, Dr. Ken- 
neth Hoy^t, at a 1978 meeting of the National and. State 
Advisory Councils for Career Education: 

Even at this relatively /earlyvStage, it 
does not seem unreasonable -to consider a 




straji^gy that calls for identifyit 
least one school system in each State" 
whose .career education efforts produce " 
evaluation data that could be submitted 
to HEW's Joint Dissemination Review P^nel 
(JDRP) for approval and subsequent inser-. 
tion into the National Diffusion Netwotk 
(NDN). This is an important strategy for 
two basic reasons: (a) it will result in 



10 



wide national publicity for dareer edu- • 
cation effort-s that have proven themselves 
to be^' effective; and (b) other school 
systems can, in later years, use ESEA 
^unds for adopting models of career edu- 
^ cation in the NDN network long after the 
brief five-year life oF'Public Law. 95-207 
has ended. ^ * • ^ 

To implement this strategy, state ^arepr educati,ofi 
coordinators will need to assume a leadership role 
in identifying outstanding local career education 
projects. This will include encouraging •and assisting 
sucTi projects in preparing submissions to the JDRP, ^ 
and then helping the projects that are approved by iv 
the JDRP to develop proposals for submission to the\ 
NDN in order that they can obtain funding to serve as 
developer/demonstrator projects in the NDN. 



NDN/JDRP; Who and What Are They? 

At thl/s point, you might be- asking: Why all this 
emphasis on the NDN and the JDRP? ^ Will they do for 
careef: education all that OCE 1^ hoping that they will 
do? df course, there is no guarantee; but, let's look 
at the ^record, and the pr^fmise. ^' - 

NDN structure . The NDN is a nationwide system 
established to assist loc^l school districts and. other ^ 
educators in. improving the)Lr educational programs ^'"^ 
through the adoption or adaptation of already develdped, 
rigorously eval;jtated, exemplary educational programs 
and practices. Begun In fiscal year 1974 by the U.S. 
Office of Education (U.S.O.E.) ,^t:he NDN has two main, 
purposes: ' \_ 



To build and maintain a national system 
for delivering an increasing variety of 



^The NDN already has a number of JDRPr-approved career 
education-related programs and practices. 

• / ■ 



r 




successful educational projects to meet 
the needs of local school districts at 
a fraction of the original development 
cost. 

- . a 

• • To ajpsure that successful educational 

projects developed in. dn^ state are made 
available f&r consideration by school 
districts in all states. • 

To carry out these purposes, U.S.O.E. 's Division 
of Educational- Replication contracts with two dif- 
ferent groups Developer /Demonstrators (D/Ds) and 
State Facilitators (SFs). ^ , . ; 

* D/D proj ects are^ directed hy local school staf f 
members who have d^^loped over a period of several 
ye^ars (often. with the*. help .of federal funds) an 
exemplary approach ^for handling a spercifiic school 
needt^'.Bef or.e being admitted into, the ^letwork as a 
^/^''>5^^^V^P^J^J^^^ ^"st have been approved by the 
Jbinti^jWsseiffinat'ion- RdA^ew Pa^^^ (JDRP); As of / 
April 1980, 245 local projects had received approval 
to disseminate from the* JDRP, repre^nting over 57% 
of those originally^ seelcing approval. Of the 245 " 
JDRF-approved projects, 138 are, receiving dissemi- 
rjation fund^ as D/Ds. Their primary job is' to . 
secure the adoption', of their programs In other ' i 
staties^ At least 50 percent 'of the pfiTtentiai adftp-^ 
tidns they obtain aiiifeally must ^ take place ini a 
state other than their own. 



NDN Facilitators, the othe'r group funded by^USpE 
to carry out NDN activities^ are pr'imatily respon^ , 
^ible for helping school districts in their owrL/ 
stat^.or fegion adopt.-^ adapt cije exemplary projects 
that? h(ave been d*eveloped in* other states and approved 
by the JDRP. They- are also encburaged to, promote 
ex^mp^lary projects that have been developed' witl^in 
'their owfi state. T 

A' ^ ^ V . 

One of the^iggest 'accomplishments of the NDN is ^ 
its redord of cost-effectiveness. The federal govern-^ 
mentis investment t{| develop the''*138 NDN prefects 
totals almost .$66 million, wi^th the median cost being 

8 - V ■■ 



$248,642. These projects are being ±ns^tai.led in 
local school districts through the NDJ^ at a cost 
to the federalCgqy^nment of approximately $4,000 to 
$5,000 each. The median per-pupil cost to a school 

V. district to adopt an NDN project is $12. This effec- 
tive dipseminatJLon of quality programs also helps to 
do away with the^ all too prevalent and wasteful 
'ireinvention of the wheel" every time an educator has 

.a problem requiring resolution. . 

The cfirector of U.S.p.E.'s Office of Evaluation 
and Dissemination (OED) , which is responsible for the 
NI5n, stated his vision for the expansion of career 
education programs in the. NDN at a 1979 NDN Conference. 

Enlarge and broaden^ the repository of 
testec|^ practices and methods. Postsecon- 
<3 dary options are needed to move beyond 
the early childhood and K-12 range, plus 
there' is a need' fi)r additional programs ^ ^ 
ill special -education, vocatioi^l educa- 
tion, and career ieducati,on. ■ . f 

JDRP structure . The NDJJ ik a federally sponsored 
approach to spreading exemplary education ft^ograms 
and practices throughout t:he nation-. The validation 
medhanismfbr the Identification of programs to be 
included in the* NDN isthe JDRP. Basically, the JDRP 
reviews programs and products proposed for dissemi- 
nation 6y federal education programs to ensure that, 
claims of; effectiveness are backed by soiitid evidence. 
Its basiq purpose is to provide a quality control 
mechanism of federal education dissemination acti- 
vities. The JDRP is not a component /of the NDN. 
However, only programs approved by the JDRP are eli- 
gible to receive Developer/Demonstrator grants through 
the NDN. Ultimately, then, the JDRP functions 4s a 
clearinghouse, a legitimizing agency, an indicator of 
federal educational priorities, a quality" control 
mechanism, and a target criterion for program d6vel- 

} 6p&^^ . ■ ■ _ • . - , : 

^he 'JDRP wasf established in 1972 by the Education 
Division of the Department of Health, Education, and 
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; ... ^ 

Welfare. The panel has 22 membeTs^ eleven staff 
from the •U.S. Office of Education and the same num- 
ber from the National Institute of Education, 
appointed respectively hy the Commissioffer of Educa- 
tion and the Director of NIE, Panelists are selected 
on the. basis of their ability to analyze and under- 
stand evidence of effectiveness,'' Panelists are, of 
course, expected to base their decisions on personally' 
unbiased and methodologically sound judgments, 

JDRP review seeks to set standards of effectiveness 
and 'documentation for educational products and programs. 
Before NDN support can be provided for dissemination, 
suitable evidence must.be provided to convince the 
^ panel jnembers^ that each of these qqestions phas been 
answered ^ / - 

©; Is there evidence that something happened? _ 

o Is the evidence credible? f ' f 

o Did something happen often ehough that 
effects as large as tho§e observed would 

, be likely to happen again? • . * . 

.... . . . > , 

o Did something happen big enVugh that the 
effects are educationally ^significant? 

© Did the something which happened often 
enough and big ^enough also- occur under 
circumstances that are likely to be repro- 
dticible? . 

o Can what happened be "a 1 1 ribih^ed to the' 
treatment? 

With these criteria, the JDRP has tri\d first to 
define effectiveness, and second to encourage use of 
professional/judgment. The emphasis is on persuasive 
or compellink evidence rather than rigid cutoffs. 
The Panel se^ks reasonable evidence on the criteria, 
hot perfection. Programs may be approved for which 
evidence on one or another criterion is not parti- 
cularly strong, if the evidence as a whole -seems 

10 ^ \ 
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convincing 'that the program is likely to benef :^t 
otl^rs^if cirefuliy adopted or adapted. / 

* ■ ' ' I 

To some educators, JDRP standards are uni;|^l- 
istically high, particularly in the areas ofi^educa- 
tional significance, attribution of measured efifects, 
and reproducibility. To others, JDRP standards are 
too low or not applied with sufficient exafctitude,' 

•particularly in the areas of statistical reliability, 
credibility of the data^ 'and magnitude and^attribu- ^ 
tio^of effects. The panel t^ies tb strike a middle 
ground between w^k^^^standards "^d unachievable rigor. 
Probably tl;^ 4)est way* tp assess the j;()RP's procedures 
is to re.ad^the detailed notes of/sesslons recorded 
since/January 1976. If you hay#- an opportunity, 

-attend some JDRP sessions,* Which are open to the ' 
public. . .4 ■■ r . 



P.roblems in Career Education Evaluation " 
.~' . . -^ ^ . / 

The federally sponsored dissemination vehicle — 
the NDN |Ad the JrDRP~4.6 firmly established,* and the 
vision of widespread dissemination of proven career 
education activities is of public record. The \iext 
questions are: What Is. the state of the art in 
career education evaluation? What is- the baseline 
practice^ so feo^peak? , ^ * 

In 1977, .the U.S. Office of Education contracted 
with the American Instituted for Research- (AIR) to 
examine the effectiveneiSs of . career education by ' ' 
identifying the. "best of the current career educa-^ 
t ion programs and practices" (as called for by Public 
L'aw .93-380) Each; career education activity had to 
furnish objective evidence pf effectiveness based on 
significant differences in measured achievement 
between activity participants and an appropriate com- 
parison group or standard. Suitable evidence meant 
that eacji of these criteria had been met or exceeded: 
' i ' ■ ' " ' 

* o The intended, positive effects of the p^duct 

or program were a<fhieved\, without appi 

ciable negative side effects. 



• The positive effects were isured reliably 
^ and validly and the obtained measurements 
^were not* explainable by chance. 

■ . » " ■ .' • 

' o ' Th^J^^. is ^tangible evidence that the product \ 
^^r program alctually. caused the measured 
effects, s 

' - .... * . . . ■ J 

Ttie magnitude of the measured effects ^ 
,^is educationally significant. ^ 

© The availal>le evidence is presented ir 

a thorough, easily interpretable, anr ' 
plausible manner. ' 

■ ' - \ . : 

o There is a reasonable chance that th. 
.positive effects can be reproduced e 
where. ' 

If some- activities could be identified as wei.„ 
evaluated and ultimately approved focodissemins -ion, , 
they could serve as guides for school^districD.^ 
* seeking to adopt or adapt ef f ectiVeycareer education 
activities. . 

In addition to a search of published and' nonpub- " ' 
Mshed reports, AIR sought nominations of evaluated 
vxareer education activities f ronr knowledgeable people 
IH^the field. In alL," 394 activities were nominated. 
Evaluation reports were obtained from 257 of these_ 
activities and screened according to stringent cri- 
teria. Following tjils nationwide search, tr^n careert 
education activities were recommended for the JDRP's 
^ consider at ioti by AIR staff and the. UiS.O.E. Project 
" Officer. JJltimately, seven of the ten career e4\ica- ' 
, tion activities were approved for dissemination -by 
the panel and authorized to apply for federal funding 
support, and two of the remaiiling thre^ activities • 
were viewed as having strong potential for approval 
if more conclusive evidence of activity impact was , ' 
provided. ^ ' » 

' A key findlnb^of this riatioiial search for., exem- 
plary programs ^as that the probability of finding 
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caiietij. .ucation projects with accej . evidence 
of effectiveness is very small and that this poses 
a serious obstacle to the dissemination of career 
efSucation activities. Are good <)rograms' really 
lacking, or ar^ there simply too many problems in- 
herent in evaluating a developing area such as career 
education? - 

Tn AIR'^ 9tudy,,^the most common reasons for 
'rejecting the evaluation design or data from a career 
education activity were: the lack* of testing^for 
st-^tist^lc&l sig- -ficance of student outcome data; 
problems in th valuation designs including eithe 
the lack of a -rison g'^roup or standard, or f" 

with the manne _ which the comparison groups .wcv ^ 
formed; and^ f^dl: Jata collection instruments or 
procedures fhat could have resulted . in error or 
bias. Most of>the problems found in the activity 
evaluation^^--<fould have been avoided. 



Measurement ^ of careei; education outcomes is 
^ihajor and widGly/^^recogirtzed problem. In the p^ :u, > 
^ progrant funds frequently have purchased audits 
process reports, consumei^ satisfaction surveys 
and, less frequently, outcome evaluations. Fe-; funds 
have been' spent on outcome evaluations, and the 
quality of outcpme evaluations when they have been 
funded has qfTTen been low. One interpretation is ^ 
•tftat most evaluators are convinced that outcome- 
orientedL^val^uation should not ,b,e attempted' until . 
the pro^gjm is well-implemented, .but time runs but. ' 
Another reason might be tha^^most evaluators, as i 
• employees of the program director , have almost no 
power to insist on thd^essential elements of an out- 
j c9me-oriented evaluation. It is also true in scaie 
instances that resources have bperi limited, . particu- 
larly 'for the ex-fcensive field testing necessary, - _ 

High quality instruments for m^-isuring outcomes , . 
ara^ not' yet* genierally avai^iable, J^cause- demand is 
^recentr. Of ten, . local test deivelopment is^e only 
v\t%asonabJM*a|)p'r^ for measuririg the unique ccutent 
, j!^'nd Qbj^^^fes involved. However, failure to i=^Decify;^ 
X th^^p^SrJ©!^ acceptable as evid^ :e of 

■ • '■ V " . " ■ . 



achievement of the program objectives typically , J 

leads to unsuccessful use of jtocal tests. / . 

' , Standardized tests can be considered for i^easure- 
ment pf career o^ucatiopr outcomes only if certain 
conditions are met.' Sufficient linkage between the' 
instructional program objectives and the content and 
^k^lls ^mpled by the /instruments is critical. Unless 
^ this cQTidition is observed, the results obtainec will 
be of ^ery little- value. ' ^ 

ii. .■ • • '\ ■ ^ • 

Surmoutii ting the Problems ■ l * 

How might these roadblocks' to measuring career ; 
education outcomes be'^^overcome? One" alterijatiye ^ 
would bd for, the JDRP ,to establish a second l^veL of 
acceptability; such* as /^'promising practices/'' Hbwever, : 
the panelists are concerned that the meaning and ' 
clarity .of ^different levels of Approval wquld become J 
b'lurred^ with the result that everything approved 
by the JDRP at whatever level would-be seen as essen- 
tially similar. Tfte^istinction between "promising" 
and ".supported by evidence of effectiveness" would 
quickly be lost. Present U.S. Office of Education 
policy allows the distribution of d'escriptive catalogs 
and listing^ of .fj^j-ojects and materials Qf unknown 
effectiven|dss only if claims" or. laudatory statements 
are avoided. This policy is aimed at ^feguardingV 
the right of potential adopters to expect that progarams 
recommended by the federal government"* have something 
substant:f.ai;;^supporting thep »in th^way*f evidence *of 'r 
e^fectivene^ ^ -n ' ^ 

i. • ■ ' \ ■ '--^ * '\ 

more^ creative and de^irait>le V^tdrnative is to 
expand, upgrade, and support local- carreer edi ition 
program evaluati ons. Her© are some specifit recommen-- 
dations.:^ " . • " • 

1. Sijnoe some of the best career education 

prdgramh don^i .kave the opportunity to ^ ^ . 

• go through the JDRP review^ because fundi}^ 
provisions ^&r pj^qnning^ and irriD^ementing 
. outcome^:^:0ludtions are lacking^ the level 
of eff^ik'^^^^hat goes into planning appropriate 

u ■ - ' ' ; , ■ ■ ■ \ 



designs Yor evaluating instruotionaZ ^ 
P' ' ''^ -du.-ts rn^ career educa-"^ 
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Incentive Att recognizes the 
m evaluations. The Incentivd 
state plans "provide proposed 
3.r-oner for the evaluation of 
State will achieve the objjec- 
ce plan.i' Although the Incen- 
e speciMc funds to be set 
: local school system -personnel 
:,Jcor evaluation planning and 
rfs specify that federal funds may 
::ate leadership foj: career educa- 
or through arrangentfSn1:s with ^ 
/ . _ . ^r.vate organizations (including 

Institiicip; s of hi her education) , . in. . .conducting 
n^eds assessments c-.d -valuations." Th^ state plan 
lechanism jould be U5?ed to ensure that an increased 
l^vel of effort is --li^d to evaluation of career 
education activitie rupporte(if with Jiticentive Act ^ 
funds. r ^ ^ 



^ Co;- 
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iuppr- 
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2. Technical assiszanae should be made Hvail-- 
able t(f loaal career educators -at the time ^ 

. ' of proposal writing so that an evaluation 
design reviewed by experts \can be included 
■ in ^thei^^^oposals. ' ''"^ ^ 

If arij appropriate evarluation design, is not a p. 
of the : ' i al project proposal, it is unlikely 
that necessary evaluktion planning and implementation 
will be carried out :nce the project is funded. Such 
technical ass-istance^.Muld be supported by either 
local or state agencies ^hat^ solicit career education 
proposals, or possibiv L- / technical assistance clear- 
inghouses: «uch as t-e OCZ-fund^d Northwest Connection 
or those maintainec :.{^ rae,. National Diffusion Network 
he prtSgtaXy one Is rhe Tecljnical Assistance ^ase, C 

lal serviceiunitA that match • 
-ding technicalvservices with 
-funded projectsXneeding ser- 



consisting of five regi 
persons or^* agencies prr 
ptesent or pofeunti£_ !r 
vices) > " • 
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provide lo^al a'apeer. eduaators with 
some fools to harry out evali^ations ■ ' 
aaaeptable to^k>he^RPy. a few generic 
. aareer eduaationev^liiaiiQn models 

should-be field tested. &ith actual . " , 
. career eduaation aativ'f ties. 

1 ' ' . ' ' ■ V • , ,■■ 

One of the products of the MR study of .well- 
evaluated career education- activities, is an 'eyaioation 
handbook that^ was . developed to. encqujrage loc^ career " 
educators to apply «ound evaluation models to their 
efforts. The handbopk .contains models for evaluating: 
(1) dprriculum infusion; 02) staff development; .(33' 
product development; IjlF) facilities; (5) supplementary , 
activities; and (6) indirect interventions. Each model 
mcL ... 2s th.e steps necessary to /voijd those problems'- 
y IR- found clilracterized . local evaluations of simi- 

i> Lvitles. ^By focusiLng-x)n ways to avoid/ the, prob- 

-■e le vrriters hoped^o encourage local practitioners 

• or kdapt applic^Je* models to evaluate? their 

--2" ucation' actfvlties\ Locar career educators 
-= uvr i to build one of the eva;Luatiori* models into 
-ure roject proposals, _ana to reipoirt to the Office 
Career Education th'^ experiences field J;estinR 
s modelf , *^ " j ■ 

■ ■ ' ■ 'w ■ ■ ' ■■■■■ 

4. Career eduaatiorT^mluation^designs ' i, • ^ 
should plaae gre»tir emphasis on non- 
test evidenae.of effects.., " 

More attention could be givenTo.; eviWce of effec- 
•txvenass tha^t includes unobtrusiye ^measures. One qf 
the h ;3t \reiVP^subm±Ssions ' in. recent years came "'from. ■ ,^ 
10-ye-i_,^ 10-school trend data on number of suspensions / 

e^^pulsioris. These data showed that the treatment:. 
scrfioc_.had large decreases -where previously there had 
• ■ f- ■ ^' J ■ • ■ ' - ^ < , ■ 

' '/ > . •■ ■ ■ ^- 

- jncier a U. S.O.E. -approved ^iblisbing agreement,- • -, 
Jlympcs Pujb^ishing Company ^of^alt L^ke C^ty ' 'Ut^h ^ 
-.3 •di-trimiting this handbook aV a minimal cost. 
-- is referenced , in Appendix C. • . ^ . \ 
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been a ttend nipwrftd and whejrp co T^arison' schso^ s 
showed no decrease, '^although the were comparal 
in mother ways» . Othei: strong JDRI^ suhinislsions hc-\ 
used' school retention rates, school ciomplcf tin i 
rates, and rates of placement in ent -'^-1^. 

5, Career education evaluation 1. gvu 
should place greater emphasis c ' inv 
mediaries dis well as on svi^-'- ' 
^ ■• ■ 

There are many innovations 

student progress that could : : . 

DR?. For example, changes in r 
- -.wl. ige, sr. d attitudes are pa ^arly 
^ -ee:^ education. Increase? academ^- i^^ J 

r .3 a 'aflable associated with iinprove te: 
*. -iment i: time §s a learning resource :ul( 
.tonvi^icihg. Innovations that improve the ifi--^ . 
effectiveness of^^use of school fun^s ccurd .e 
3mitt^d.\ ^ 

' " \ . 

eciflc rielp for the Local Career Er icator 

In this monograph, a^^ecklist jp;i\ ng _nu: r en^ 
tial points fori local career ^ducat^"^^ tc. c^i .^i.:. 
in planning tfieir evaluation desi i :i_ . ;se 

In detail,, with .full opportunity ^or readers : e 

th--:: list 'trb assess, an existiiiig design 3 well 
Although/^ post hoc assessment has man; 'lljnii lxo: ^-^ 
as was found 'in , the , AIR study^ there are usH:L__ly 
spme possibilitijes. fo^'improying design:3-or r r ' ' 

reanalyzing ^t he Outcome data. Followit the r rese^ - 
^^tation of the cbfcklist, specific reconmenjdati ^ns v:ili 
be given to the Ibcal career educator n^^ardin ! tlfe • 
preparation and submission pf ""an evalua-zion ret?ort - ^ • 
to/ tne JPRP.^ Much has been learned f rem th^ AIR " 
study; ^;hese findings, will be laid out in a step-?^y- 

)roach. Thd" federal government _s*EFirresponsible 
does- not take stens/to ensure th^ir the educational 
Ls it '^disseminates are backed b}- ob^^tive 
2 of* effectiveness; the joint USOE/NIE paiTel, 
that the same criterda for fudging eviderice 
:tlveness i^^ior to federal dissemiij^tlon a^e 
to all -submissions^ ^ 
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Once thevcomz-I r - • 

* the JDRP Is spe . : 

* tO/*a section ^ i 
your ' colleaf o: ^. 

, * dally given s':i:.t! ^ 
cpntains thre^ ? ,r . . - s 
AIR stuily. l;£ Sc- 
\ ^submitted ]to J -.2 i 

* -tion project .recrroxc 
'submissions ,v7er2:; aDprc 

^ un^n^imaus vote f ::fiG 
" . the appendijc 1:: J.udes .. 
• Jikel]^ ;tD bp )c _ use "to^i 



?ss involved ' 
ou* will be i 
>Tninon quest ion ^ 
::^d ,and ,the j^nr 
-ons. FinalX^ 
JDRP submiss: 
1 the e^iact 1( 
. particular* 

all, thre^ inr^ 
by/ a unanitnc 
^1. The last 
St of, additi 
J reader. 
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irs typi- 
:in appendix 
from the 
i^^e report 
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sound evi-' 
r the obsetvBG 
lould, be a 
2cz; with' . 
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•4 ,For a caree: 
effective, th^t 
i^dence that i^ wa , in 
change in the par^^cip 
, ^continuous -and ongoing asvi^zt of any pre 
^a. key.b^ingrthe value of x :h process an 

an aid to tE-riiigement deciszonn^king.' 
"^}^^^^ evaluation desi_:5n which is directly ' 
. relat)Mj:Q the' project s::^ted objectives shoul'd be 
; yf'estab^^hed early arid base :£ne/.data should W obtained 
V prior to, or at least ^>onc rrent" with, initiating, the 
delivery of services. The :iheckaist b<iow gives -essen- 
4:ial points for locaV career edjidators to consfd^r in 
, planning their evaluation^designs. • ^ ' 



TREATMENT 



Needs 




Questions 



1/* Are the needs -£he treatment is 
I designed' to fulfill* important? 

2. ^re the needs defined in terms 
of required changes in the way 
people think, f^el, or act? *y 
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Topics 
TREATMENT (cont.^ 
^Objectives 



Quesfclons 



Pro^cedures 



3, Do the desired outcomes of the 
treatment relate to its needs? 
Is there -a reasonable chance that 
if the outcomes are attained, 
the. ixpeds .will Ue met? 

'4. Are the desired outcomes of the ^ 
Jtreatment appropriate for its 
intefided aXidlBnce and circum-^ 
st*ances of use?, ' \ a 

5, Are the desired outcome? either 
ex^icitly stated ot^ clearly '"^ 
implied?.. ' ^ 

6, Do treatment procedures represent 
an implementation of the treatment 

, objectives? 

7, Are the procedures likely to pro- ^ 
^duce* the ^i^ired outcomes under 
the intended^ircumstances of *use? 



8. Are th^ procedures direpted ^t 
'care^ education '^out comes distinct 
and not confoqnd^c^ with noin-treat- 
^' 'mei^t ' compt)nentsj' " . 



25- 



9. Are there likely to be no unin- 
tended side effects, particularly 
undesirabl^'ones? * • 

10, Ar^p the effects of the proceRires 
measurable in the short run? If 
not, can^ suitable proxy measures 
be developed? 

. / ' ' ■ 

Can the procedures fej^ implemented 
/ithln practical time and resource 
cohstraints? 
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Top 1 cs 
treatment' (cont.) 



Questions 



EVALtjATION 

« » 

Evaluation 
Design 



12. Are the procedures consistently^ 
employed by staff responsible 
for instruction? 

13. Are the derails of instructional ^ 
procedbres'^available t€h provCT^ 
adequate descriptions of treat- 
ment methodology? 



14. Are adequate .jdesc^riptive' data 
;characteriziifg the treatment 
group provided? 

Are adequate descriptive data 
characterizing the comparison 
groujp-provided? 



1^. Are procedures for selecting the 
* comparison ^group' specified? 



Measures 



17. 



18. 
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20. 



.^s the comi^rison ^roup reasonably 
companrable to the t^reatmetfe group? 

Does the comparison group use ^ 
typicAl instriiotional materials 
and procedures? * 

Are the measurement instruments 

either standardized Jtests or 

specially constructed tests with 

reasonable and v^4!fiable validity 

and reliabili^zyr 

* ■ - .^ ^ 

Are the measurement instruments 
appropriate for evaluating the 
desired outcomes? 



Topics 

r 

EVALUATION (cont.) 



^jchedule 



Data Pxo- 
cessing 



Stal^istics 

7 



Results 
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22. 



QjLiestlons 



Are the instruments administered 
, to treatment and comparison groups 
according tx^the same schedule 
and under the same coriditions? 



Ace the classification and grouping 
o|^ raw data for processing appropriate 
for the methods of statistical analyfeis 



usecf? 




Educatiohar 
\ Signifi- 
cance of 
Results 



Consistency 
of Results 



V. 



23. /A^ the statistical; analyses used 
^ apprbp^rifete f^r ^he evaluation 

design? 

24. Is a di'scussion o^ the meaniug 
tfie. results provided?^ ' ' 

' • ' y 

25. Is a plausible inferpi:ef:atiori of 
the rek^n for 'tfi^cresul^/s given? 

26 diiferehce in performance 

-o^ Llig -ferea^tment group ig statis- • 
tically signH^f leant in comparison " 
with an apipropt^ate comparison 
group or standard, has the eduea- 
tio^il significance of this differ- 
ence been explained? , 

27. Do the evaluation data give promise? 
of replication, or are they verified 
data fi;om adequate replication(s) 
le treatment? 




Explanation-^ f Checkliat Items . 

Ne^ds, objectives » proaedures . Evaluation reports 
submitted to the JDRP are expected to provide an adequate 
description of treatment procedures. Such information 
is necessary for potential adopters to 'decide whether 
to try out 'the program in their^ school- districts based 
on^^ adequate uncierstanding of the career education 
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program and the factors responsible fpr iti^ successful 
; operation. Ihftjrinatioa should include the program's 
' conceptual framework and mfethodology for identifying 
priority needs, its major golils and objectivedf, its 
instructional procedures, and its management plan> 
Information about ^he number'^tyf^students j/nyolvecf , • 
/ their ages,, grades, and other relevant demographic 

characteristics is also required. Info^-fnation on - ' 

* teachers include^ relevanjt a^pects^ of ' their training, ^ 
qusrlif ication^, an^ ;.,jer4;'ence. Information on the^ 
context includes demographic -^ltf^^3^pc of the', . , 
locale, rel^ant^a^pects '6f the scTiool system (e.g., ^ ^ 

\^ize and finari.0<^l -status) , and t^fe his-torical back- ^ \4 
ground of' the program, ft is 4lso necessary that , 
prdgram materials be "lavaicjaljle *f or' -dissemination. ' 
An adequate "description gf the treatment implemen- 
tation hel'ps'rature users to implement .tl^e/p^ ofer am • -x. 
ptop'erly, . '-"^ V ^ \" C -* " .- 

Since ^financial consider^^feLons -►are key issues fot: 

* school districts "z^lanning adoption, it ft necessary* 
to include c^st information.* Although differences'' 

^in tot^l costs anJ in cost categories exist among 
programs and locations,, it is still important to 
provide kccuratJ^ start-up and-f-^i-acurring budget guide- 
lines for admmfe trative, cieriiial, ^teacher, ^and 
teacher-aide sale ries, ^matetials, equipjnent, facili- 
ties, consul tatfit: , services, t^ravel, testing, aad 
evaluation.- Total per-student\ costis must be presented 
so that any add-on costs for adopting the career 
education program aire clear. 

Evaluation design . A gpod evaluation design must 



h£ ^ble tQ*fsuppprt strong inferences , 'particularly 
Regarding I ad sejrtions that the treatment as ifnple- '\ » 
mented caused the* effects measured, and that the par-- 
ticipants are a representative sample of. the entite^ 
defined auHi^nce. In principle,/ the design best 
supporting strong inferences is one* in which two 
^subsamples"^ of participants are randomly selected: 
oril subsaitiple (the treatment group) participates In 
the caree;r -education irfiplementation; the other (the 
control or comparison group) ddes not. In addition, 
outcome tneasur^s are administered both before (pretest) 

• • 'v ■ . - . : ■ ■ 
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and after (post test) the treatment is implemented. 
The performance of the comparison group is used to 
indicate what the participants' scores would have 
been without having been exposed to the treatment. 

In practice, it ^^rarely possible to use this 
randomized pretest f ^p ttest control group design. 
There ajie various reasons for this. There may not 
be enough persons available to construct two groups. 
It may not be possible to pollect all the needed 
data from the nonparticipants , who have no incentive 
to participate in the evaluation. Under such, circum- 
stances^ compromises are necessary. For example, 
the project. director of ten will have to use a non-. 
randomly selected group of nonparticipants, chosen 
by their willingness to cooperate. If so, an effort 
should be made to provide evidence (by comparing 
the pretest performance of the two groups) that the 
two groups -are not inherently different. Remember, 
tixe strength of the inference depends directly on 
the strength of the evidence provided. 

Other appropriate comparison grotips or standards 
include: (1) national norm groups; (2) local norm 
groups (norms for locally developed tests); (3) 
trendVdat^ (4) experimental expectancy (gains of 
the tibeatment grbup are compared with expected gains 
as indicated by research) ; (5) "time-series samples; 
and (6) alternative treatment groups (several treat- 
ments admin J/jStered, with the possibility of a control 
no-treatmenb group used al:so) . ' ; 

The strength of the^inferehce is also directly v 
related to the number of persons involved in the 
s'ttidy^. The more participants, the greater the cfiancie 
of detecting real differences between treattnient and 
comparison groups a^fid the more generalizable the 
'results. However, it is often difficult^i^ involve 
large samples in the treatment implementa^on and in 
the comparison groups. Thus, in general, 25-40 
piersoris per group are probably ^equate.* If a number 
of persons drop out after the pretest, the pretest- 
scores of the dropouts and non-dropouts should be 
compared to ensure that no significant factors (e.g., 
ability level) caused the attrition. .^ Program effec- 
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tiveness should be measured only on the basis of 
those who. took b^th prer and posttests. 

' ■■ ,) ; 

Measures . Although the problem of selecting an 
adequate evaluation instrument is common to all 
kinds of educational evaluations, it is especially 
critical in the area of career education evaluation. 
The difficulty in identifying appropriate instruments 
'to measure career education outcomes is the most 
frequently mentioned problem in career education 
evaluation. The issue facing-local career educators 
is to determine whether valid and reliable instruments^ 
are available^ to measure participant change, so that 
data can be trusted. Measures are acceptable if 
they are either standardized tests or locally con- 
structed tests with reasonable, verifiable validity 
and reliability data. 

With regard to locally developed measures, the ^ — ^ 
major steps to follow are to: (1) select the types^^ 
of instruments needed to measure the outcomes ±det(^ 
tified; (2) draft appropriate measures; (3) tr/out 
the drafted items on a small gtoup of students similar 
in characteristics to the target group (a mini-pilot 
test); and finally (4) revise and refine the measures'. 
All measurement techniques have some drawbacks in V 
all of this following criteria; what is important in 
designing measures is to maximize the levels of 
adequacy. Let's discuss how to accomplish this. 

• Objectivity . Objective methods are those 
that yield similar scores no matter who is doing 

the scoring. In general, paper and pencil tests, /j? 
checklists, and rating scales are more objective 
than performance tests and observations. It is 
necessary to establish scoring rules that facilitate 
clear assignment of scores to each response. For a 
less objective test ^.g., an essay), this may 
require developing a key that gives scoring rules 
and examples of typical responses and their proper 
^ scores. * 

• Reliability . Reliable measures are those 
that yield constant scores relatively free from 
chance variation over time. To improve reliability, 
it is important that instructions and testing con- 
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ditions be the same for all persons; that practice, 
or sample items, be given if possible to avoid 
effects from unfamiliarity with the type of measure, 
being used; and that several measures for the same 
objective be used rather than one,. ^ 

S^- Validity ,- Valid measures are those that are . 
closely related to and broadly representative of 
the outcome being measured. Measurement tecfmiques 
that are relatively objective and reliable also tend 
to be relatively valid. However, additional assurance 
of a measure's validity is obtained by constructing 
a logical rationale for each measure used (to see 
bettier its relationship with the desired outcome) 
and by providing sufficient, measures of each important 
outcome. If several independent measures of the same 
outcome produce highly similar results, the measures 
are likely to be acceptably valid, ( 

. • Efficiency , ' Efficient measures are those that 
yield reliable and valid scores at a low cost in terras 
of money, personnel, and time. In general, this means 
that the measures can be administered to groups on 
a single occasion and under normal rather than con-.' 
trived circumstances (e,g,, an ordinary classroom 
setting) , Measures that can be scored and processed 
quickly and easily are more efficient than those 
requiring more time and expertise, 

• Non^reactivity , A non-reactive measure is one 
that does not unduly influence the behavior of the 
person to whom it is being applied. Relatively non- 
reactive measures include routinely collected records 
and observations-r-gor example, observations /of the 
frequency of certain behaviors. Such teqhn^ques may 
not be as relevant to career education measurement 
as tp other areas,' but it is important to be aware 
of the level of reactivity and to try to minimize it, 

■Schedule . Procedures for conducting the evaluation 
are: scheduling, identifying and training administra- 
tors, identifying and orienting participants, and 
administering instruments. 



A detailed schedule for training administrators, 
orienting studentis, and collecting data should be 
prepared* It is best to collect data in as short 
a time period as possible to minimize disruption of 
regular school schedules. Rooms of adequate size and 
appropriate facilities should be reserved. Schedules 
should be made available to all participants to' . ' 
facilitate coordination- efforts. ^ ^ , . 

Different types of people might "serve as test 
administrators. Regular teachers can probably be 
used if the evaluation focuses on paper-and-pencil ^ 
tests. If other types of measures are used, siich 
as performance tests or work samples, it might be 
preferable to hir;e- oqtside pers^ons from t^he, relevant 
career area. Depending on the ta^sks tt> be performed," 
some training of administrators might be needed. ^ 
Training can employ a cbmbinktion^of approaches 4.n^ " 
eluding written" materials, ''workshops, and practice 
exercises. ' ' . . - 

■ . ■ ■ . ' .... ■■ ' ., ' 

Participating students ma3^^reqi^re some orientation 
to the purposes' and 'proqldures .involr^ed^Mn the eval-^ 
ua^tion. Orientation shbiild* be' unifaVm for^ir^studerit 
and' can include both ,\^itterf and orally: presented ' .\ 
material^.^, . - ■. ' ' - ' , 

t ■ ■ ■ ■ . . % . • ■ r.-.-- ■• 

. Data proc^^3ing"an8^ st^tistlQSw « ^The 'dhoice of : . ^ 
appropriate proce^ssing Jnd^haaysi| fi^fdcedure'^^iVt^pends 
largely bri the. measu'^^es u^6a> the' evaluation d^^^^ .: : 
and the resource^§ availalJle. I tH=s possible tl^^^ 
of the ^ata prQcessing;wili be ^one manually^ espe^^^^ 
cially whf n^saraples are siqaii; ^'d' iffe^Wrc^^^ 
Often, hand-SGprtng.^in^Ql^^ about 
the adequacy of i rejgpons^ cyr^ t^rans^Stli^ -a^ coipiex^ 
respbnse into^.a ^singlfe'' scqre;.? ' Thls^wQ^l^ b^' ^true^ 
f6r:examp^^e, ,'^n evaltiattng a Wdj^:^ San^|e telate^^^ 
the 'instructional pr;Qg.ram . Ce.&5;, coApos^ing an ^dexLua^ 
^Letter requeStiri^ a Job intetvi^^)'. . .T. ■ 

g-^Wh^re such- j\i(fgme^t^ls^need^^^ several 4)oih^s ' 
shpuld be considered T'; Firsfi/'isddes: the - scorer ;under- 
stjgrid /the .crrt^ and .^ppS(^ them' ^ over 

'Secoiid, has inforW.ti-b^^^ 
pant^heen Concealed, to he>p ensure scorer . obi ectivity? 

•26 ^ ; . ^ y ■■ :■ . ' 
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Third, would two or more scorers assess the same . 
response similarly? To answer the^e questions, it 
is desirable to score a random sample of responses 
twice. If results are ,inc6nsistent, it would be ♦ " ^ 
wise to retrain the scorers and restore doubtful 
items. - 

Often, a computer can be very helpful if resources 
permit. The evaluator's job is* to have data recorded 
onto computer cards or tape and to determine what 
information the computer, should provide and how it 
should be presented. A computer programmer can then 
write*^r select an appropriate package, and the com- 
puter will provide the results.. v - 

■* 

The next step is analyjzing the processed data. . 
As a minimum, evaluations<'Of career education activi- « 
ties are expected to include the following statistical^ 
data on both treatment groups and comparison groups 
or standiards; _ (1) measures of central tendency; 
(2) measures of dispersion; (3) attrition rate; (4) 
pre- and posttest distributions (when appropriate) ; 
and (5) obtained level of statistical significance.! 
Tests of statistical significance are used to determine 
whether differences among groups tested greater 
t^an those that would dccur by chance alon^. 

Results . Criteria include the statistical signi^ 
ficance of the results, the consistency of the r^sulte, 
and their educational significance. Evidence for a 
career education program's effectiveness must not rest 
solely on statistical significance, which can be ob- 
tained for very small effects by using sufficiently 
large sample sizes. Statistical significance can be 
and often is achieved with a difference between means 
of one raw score point or less. Thus, do not. assume _ 
that statistically significant results alone,, demonstrate 
that a career education program is worth maintaining 
or replicating. . , 

Educational significance has to do with the meaning- 
fulness of what is observed; Are differences between 
the treatment group and the ^comparison group big enough — 
large enough — to suggest that the high€?r-s coring group 
can do something important that the others cannot do? 

27 
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In career education, for example, do treatment group _ 
students show enough increase in knowledge about occu- 
pations, competency in basic academic -skills, good 
work habits, and personally meaningful work values 
that they are likely to make better choices?- In 
other words, is the observed difference large enough 
to be worth^the effort it took to^chieve it? The 
size of the treatment impact can judged by past 
experience as reflected in the research literature 
indicating the expectW frequency, direction, and 
size of post-treatmer^ differences between treatment 
and comparison gr9ups on the measure' in question. 
One rule of thumb for evaluating the educational 
signifi<:ance of a treatment is. that the obseryed 
^posttest scores exceeded the expected scores, either - 
comparisoneroup scores or scores determined on some 
-other basf^ by at least onerthird of a standard 
deviation. In addition, the cost of the treatment 
can be compared with the cost of other activities thae 
.produce the same effect. \ 

'-. ■ * ' ■ 

' 'The Nitty-Gritty of Obtaining JDRP- Approval 

^ The purpose of this , section is to' demyst i,f y the / ' 
JDRP approval process, for local career educators and 
to spell< out the procedures involved in getting a / 
10-pag6 \iritten submission up before the panel. . 
,A fictitic^jjs panel deliberation will then b'e presented 
to demons t^te hoy the panel functio^is during one of 
its session:^. * 



Writing th6 ^ Submission 

A- very valuable ^resource that can be used in pre- 
paring submissions is the JDRP Ideabook (teferenced 
in Appendix C). Submissions should contain two key 
substantive parts: 

o Describe the project briefly (3 to 4% ' . 
pages) including the following key ' - 
points: 

28 . ; " 
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- what, the 'project intended to do 
(the goals) 

,7/^hat the^project did (the treat- <; 
^ ment) ^ , 
. % 

- who the intended users were (the 
target groups) 

- in whai ways the project worked 
(claims o^j^f f ectiveness) ''^ 

- what the project costs were — 
divide this into initial instal- 

/ lation and maintenance 

Present evidence of project effectiveness 
or evidence to support claims made above 
(6 to 7 pages). Fpr each claim made/ the 
following points must be covered: 

- describe exactly what you, measured 

describe exactly how the measuring 
was done, (e.g., tests, observa- 
tional ratings i skills completion 
checklists) ^ 

- show that" the measures used ^ 
' actually measured what* they were 

intended to measure (measures * 
are valid) 

- show that the measures used 
actually produced consistent * , 
results with repeated use 
(measures-are reliable) 

- show that the measures were 
made by someone who had no vested 
interest in the success or fallu:^e ( 
of the project (the procedures . 

are unbiased) 
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■ . • . . • 

- show the treatment produced '* • 
# some result or cjiaf^e that is . ' 

different from what wbuld have ' 
/ happened Jwith no treatment ^ . . • ' 

- ^ - ■ show that the change achieved * • * ' ■ . 

- is truly meaningful to the fi^ld ^: ' - 
of career education ' 

, « • ■ 

* show that the change is due to a . 

the treatment and not to any 
extraneous, factors 

; - show that the change can reason- 

ably be expected to occur among . ' 
other users, not just thosd tested ' ' 
±n the project (project is general- ' , 

izable) ' . 

- show that the charige can be repli- 

cated elsewhere with comparable " * 
effects, considering cost, facil- 
ities, staff ,^etc. (project is ■ . ' ■ 
replicable) . ' . . • 

• ' • - 

A sample format is included in Appendix A. 
Since the written submissions are the principal 
basis of the panelists', judgments. . it is impor- 
tant that this information beclear andy. complete. ' < ' 
Theimaxlmum Permissib le length .of ^materfa1 «-n 
each submission is ten single-spiced typed pages, ° 
Including tables. The most imp6r1:ant section. *.of ./ 
course, is the summary of the evidence of effective- 
ness. Jlowever. do not neglect the brief but specific 
descr.iption of the product or practice (including 
goals\ or targets of . the process) arid data bn program ' 
costs so that the panel can consider the educational 
signlfici^nce relative to the , effort and costs required 
for adoption. The panel is concerned that the pro- • / 
ducts or, practices whose effectiveness is to be re- 
viewed are clearly identified, that common implemen-^ 
tation problems are described, and that full start.-ra^.' 
and operating costs are presented so that s^hoois^ ■ 
interested\in selecting the item will do so without " 
migunderstaWing^its costs, this' information provides V 
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the basis for -the one-pagtH^roject suinmaries 
^Included in Educatio^S^Trograms That 'Work (refer- 
enced in Appendix C) , the federally-founded publication 
containing descriptions of all JHRP-approved projects. 

. Includeci in Appendix B are three sample JDRP^ 
submissions for the career educator to study in " ' 
depth. 'Their credibility is established by the fact 
that the JDRP approved -^one of them unanimously, 
while the others received just. one dissenting vote 
each. All three were approved in 1978. * 

Action Steps in Processing the Submission 

Prior to its teview,^'^^^^O.E. 's Office of Caree^f^ ' 
Education requires; that the submission, secure state 
approval. Then,^ the OCE staff will review and ini- 
tiate dialogue with, you conceriiing the submission. 
It will b^ an OCE Istaff member who will eventually 
present the project for JDRP api^roval. 

If'^thp project! is not approved by the OCE staff 
for submission tcj the panel, ifci^ simply means that 
based on their advice and^expertise, the project is 
unlikely tb be approved, ^view the project racord; 
and evaluation data. Stifxfy-ltir^OCE staff? critique:, 
and then make the decision to/ continue or discontinue 
with the effotc' /to,..<^ffibmit to. the JDRI .: Many progranzs 
have had impact /i'np^e local educational agency yet 
could not be applicable in other locales. ^ 

Once OCE i4; satisfied with the submission, a ■ ^ 

memorandum of transmittal is prepared and signed by 
a U.S.O.E. Deputy' Commissioner stating that the sub- 
mission has /been pre-reviewed and, to the best of 
the Deputy .Commissioner's knowledge, is fa<?tually 
^correct and t!jat ther products or practices are soci£-ly 
fair and a^par^ntly ^not harmful. This is the^offici-l 
notificatl/on of a project's readiness for^ presentation 
to the JDRP by the OCE. When this memorandum is signedi 
it certimes to the JDRP that all stages of investi- 
gation have been completed* « 



Fifteen copies of the transmittal ,meln(/r^ndum and 
each submission are forwarded to the JDRP'^ Executive 
Secretary, who schedules the" JDRP meeting. Although 
the JDRP is made up of eleven persons_^from_each 
agency (U.S. O.jET. and N.I. E,)i the meeting requires 
only seven (gt least three froirf each office)' to be 
present in order to have a quoiW. Panel members 
require a minimum of f ive* workijig days to review the 
subm^sions prior t5 a meeting.'' Typically, the panel 
•will conisider thre^i siiBiiissions at each meeting, with - 
each submission r4cei-/ing ira to^ an hour of :he panel's 
time. An OCE staff member T7iir represent me project 
at the meeting, u£-aaliy accompanied by the local project 
direcftor and the project ev-aluator.^ 

< ■ 

Activities During and After the Panel Review 

The bulk of the time in ay^anel teview is spent 
on questions^for/ clarificat^n that may be - t to • 
the Program -Office , on tc he project dire -or - 
and evaluatbr, by :he pai^e. .^embers. A vot- is taken 
pn whether there Is 'adequate ^videnpe of ef-'ective- 
ness. Voting is by .closed ballot, but the -rotes of ' 
< the individual' panel member: are recorded b-/ the 
Executive Secretary after e-::;ii vote has been completed; 
Panel members disia/ualify themselves from vc ring on 
submissions ra'ade by program^ in which'.they are directly r 
involved. A simple majoritj; is required for a 
panel detisibn"* than the evidence of effectiveness is 
convincing.' A tie vote is a decision th^t zhe evidence 
is not pony incing. A majoriry vote caWies ;ith it 
not only -the honors and national recognition but also 
the respon^bilities of being added to a select list 
of exemplary educational programs. The panel's decision 
is atinounced- publicly ^.^ediately following the proiect 
review. ^ \ ' \' . 



, Products* or practices not favorably reviewed" 
by the panel may be/ resubmitted" at any time. The. * 
panel makes every/^fort to clarify weaknesses, and 
makes recommendations for a stronger resubmission. 
If a project director feels that through resubmission 
he or she can satisfactorily^answer the panel's con- 
cerns, then resubmission is ^advisable. If not, do not 
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resubmit. The panel itself often recoraaends resub- 
mission following comments on a .program's defined 
deficiencies (naturally, approval cannot be guaran- 
teed). The minutes from each meeting are distributed 
to all pan'el members /"OE Deputy Commissioners, and 
N.I.E. Associate Directors; copies can be obtained 
by those whose submissions wei^ej^ reviewed in the 
meeting. r 

w 

The following is a fictitioiis but descriptive' 
account of a typical panel meetinjg: 

The panel convened* at; 9:30 a.roi. ir^ Federal 
Office Building 6. The six panel members 
and the chairperson sat along one si.de of an 
oval table. The panel's Executive SecretarV 
was also present to retord'the proceedings. ^ 
The first project in front of the panel 
a high school career education activity titled 
Project AMERICA. ^The chairperson invit^d^'the^ 
representatives of the project and the OCE " 
program official to, join the p^nel members at, 
^ the tablet. ' " ' ' ^ 

The OCE program official fiyst introduded 
^ all, of the participants, including the director 
of Project AMERICA ahd its evaluator. The \ . 
chairperson then* proceeded to summarize the 
the- submission, both to refresh the panel 
members' minds about which project was being 
. reviewed and to describ.e it for all of the 
visitors at the meeting. The chairperson had 
^apparently gone thr&ugh the submission quite 
extensively in advance, highlighting portions 
that were particularly important. 

The chairperson then opened the session 
for questions from panel members. For* a few 
minute^, silence prevailed since no one asked 
any questions. Finally, a member of the panel 
sa^d that it was hard to* pick up from the 
subijiission^the nature of the program activities 
that would be exported* He asked: What did a 
teacher actually do in the classroom as con- 
trasted with what a teacher did in the control 
/ • 
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classroom, and what\s in thg instructional ; - 
materials package thafc would be exported?, C 
The Project AMERICA^irector' sp^t about five 
minutes summarizing the main pr^cfedures and > 
products involved in the program. 

' - ■ ■ ■ ;\ . ' . . : - • 

Several panel members^ then asked some ques- . 
tions about the evaluation, which' the project 
evaluator answered. One was a simple clari- 
fication question. Th^ others ^nvolved^^ 
differential ef f ectiveness\^£^he ^rogrW' 
at <Iifferent grade^ levels, ^ whether' thfe program 
had achieved results that spoke to the sus- 
taining effects question, and/whether the ' ^ 
effects might hold up with other not s^o • 
similar student papulations. At the conclu- 
sion of this dialogue, one .'of the panelists - 
suggested a caveat to dissemination of this 
program: ' data presented are restricted to 
-white^ middle-class students. 

\ AfterVa ^pause, the chairperson askjed ^ • 
whether the score 'dif ferences between treat- 
ment and control students were large enough 
to be educationally significant, referring 
to a table in the submission containing the 
results from administering the primary data 
collection instrument. The instrument had , 
54 items and the mean score difference between 
treatment ali^d control students on the»p6sttest 
was 5.96 points. The chairperson asked the 
prbject eyaluatbr to comment on what ^ mean 
difference of 5.96 points truly meant in terras 
of (educational significance of the instruc- 
tional materials! The proj_ect . evaluator 
responded by pointing out that such a difference 
was meaningful, given the fairly small amount 
of time that treatment students were exposed 
to the materials and the low cost of the % 
materials. ' 

The chairperson then asked if any paneK 
members had , further questions. One or two/ , 
other clarification questions were asked of 
the project director. When no adcjitional 
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questions were forthcoming, the chairperson 
asked the pan^^embers to vote. Each panelist 
had a' ballotvwith the project 's"^ name pre- 
printed o:^ it on which he or she voted "yes" 
or "no" for disjpeminatiori'. Since each member's 
vote is eventually made public (in the tran- 
scription), the ballot also contained\a. place 
for the name of the panel member,. - 

The chairperson collected all of the iJallots 
and 'counted them. Some suspense was generated 
while something that lookied fike recounting 
the votes took »lace. It turned out that 
spmeone on the'^anel had marked the ballot for 
the group that-was Opposed to present its 
project next instead of. -the Project AMERICA , 
ballot. After^ that error was corrected, the 
chairperson announced the vote: 6-1 approving. 

^ * ' it ' . f ■ " 

r ^Questiops You've Always 
: Wanted to Ask, Etc. 



How does one separate state validation, the JDRP 
process, and the NDN? 

Ans? Those are three different activities. TJhe , 
state validation procedure is not required 
^ by the JDRP.' It is^purely a function of the 
states. Some states do require stat^ valida- 
tion before considering a JDRP submission; v 
Other states go straight to the JDRP. Approval 
' by JDRP means that a federal progranj^ffite 
may disseminate -the prorject. To receive 
federal dissemination funds from the NDN as 
''D/Ds, those projects-that have cleared^he 
JDRP submit proposals in a competitive^ 
bidding process^ ^ 

Does the panel take into consideration the type of 
project represented in the submission, inlightqf 
* tlie voids that exist in the "NDN? 
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Ans: At the .JDRP levels, there '^i^o prioritj'' 
list. The iydgment -of .title part^lists'is 
based on the evidence presented. Did it 
. hold up or didn't it?, In terms of projects 
. submitted to the NDN^there is a new situa- 
tion tq look at. . Within the NDN submission 
system* there is a priority list. There 
y ' ^ is. a move on to enlarge and broaden the 

. \ repository of tes^MTpract'lces and methods. \- 
« For example, to move beyond -thjB early -Wild-- 

hood area, the NDN is opining up. k wider 
- avenue for. postseftpndary opt.ipns as,well v 
as for additional' programs inJsp^eciai edu- • 
cation, vocational eduq^-t ion, -ana career 
education. '-/ ' .'. | 

i. Is -a ifrogram that^pa^seS >the JDRP aurhorized-^tfr " 
appear, in Educattohal Programs That Work ?-- <' ' . 

J ^Ans:^fyes, but thatt does? not, automaticah^x/Wlude 
,.- NDN funding. Only, about 50% of the programs 
- ; ^ that ^appear^ in Educational Programs That W nr> 
receive ;i1)N funding for dissemination. NDN ^ 
funds,- like other federal funds, are provided 
• for in the annTial' budget. The level of funds 
^ . . ' available for .disse^iination purposes may vary 
from year to year. 

^. Is there any predictability as to which seven of 
the ,22 members of the panel will show up for any 
given , session? .' ' • 

^ns: -No. The Executive' Secretary «f the panel 
knoy/s the nature of- the submissions before 
he calls together a quorum of . seven- panelists.. 
, ^Members of each^ panel, are different as selec- 
tion is basically rotational. Some members 
are Selected on the4)asis of a particular 
expertise which could assist the pane Vs ' 
review. 

5. Is, there some wa^. of knowing who 'is more. stringent • 
and who is less stringent among the members of the " ' 
panel arid of being able ahead of time to influence 
which, seven members will make up your quoruiq? 



Ahs: Certain configurations fejf seven panelists 

may be more stringent than others. However, 
the structure pretludes any outside influence 
being brought to b6ar on the composition 
.(of a pa:rticular^\-qtiorum. 

6, If there vis a resubmission, are you assured that 
the panel will*^be reconstituted to include the 
same members who served when the original sub- 
mission was teviewed? , 

Ans: No," However, the program pfficial, when 
introducing the project director and the 
' y- project evaluator at the second session 
'of the. panel, is at liberty to say that ' 
this is the second time around. "They Wjsre 
asked to go back and get such and such type 
. of data and it can be found on page 7 of 
the submission, etc," The second panel, 
which may include a member or two from the 
initial group, will completely review the 
earlier panel's concerns in the context of 
the submission, . ^ 

5, ■• . , ^ ■■ . ■ '. 

1m Do project staff have any influence oVer.the order 
• in which the submissions are reviewed in a given 
session of the panel? a 

Ans: Normally, you are scheduled to appear in 
the order in which your submission^, has 
entered*into the system. However, if you^ 
are scheduled to appear last, arid — let's ; 
say-Tyou have a plane flight out of Washing- 
ton, D,C. at 11:30 that morning, , a request 
to the panel will usually result in an 
adjustment being made 'The Ranel tries to 
be as. flexible as possible, 

- . ■ • - ' 

8, How oftfen does the pariel meet? 

^ ' ' ' 

Ans: There is no regular schedule. It meets 

whenever as many as three' submissions have 
,been received. The panel has b^en known to 
meet to review just one submission if others 
have not com^ in ^recently. 




9. Looking at 10 pages of written material and data 
tables, how do the panelists know that what they 
are reading really went on in the prpject? After 
all, the JDRP does not require state validajtion 

. ^ in which site visits are made, and the panel 
members do not make visits themselves. 

Ans: The panelists rely on *t he credibility of 

the project representai:ives in that session, 
and even more on the credibility of the 
written submission. The panel, of course, 
relies on the federal program office vouch- 
ing for the submission, too. 

■ . ■ . ./ . .n.- , 

10. Is it absolutely necessary for a project to have 
government funding (state or federal) l^efore it 

can be considered by the JDRP? ' '^o ' 

. , . %' . • . • 

, Ans: Jhe JDUP has recently. changed its position 
/on this question. Historically, only 
federally funded. projects qualified. Under 
the new rule, only for-profit corporations 
. are excluded. / 

11. If I were a practicing school administrator or 
superintendent, how would you respond to me when 
I argue that because of JDRP /NDN funding, I am 
losing the services for anywhere from 45 to 60' 
days of the most qualified persons on my staff 
as they travel around the country, while the 
taxpayers^^of my own district *are not receiving 
those services? 

■ ' ^ ' ' ' / ' ■ ■■' ■ 

;Ans: , Some very good projects have' never gone 
thr6ugh the process because of that very 
issue. What it really comes down to is that 
if you. look at the history of education in 
this country, you have a number of problems. 
Number one: in education, pej>ple keep rein- 
venting ihe wheel. To answef your administra 
tor, yoJ omild say that w^ have coihmitted 
our efforts to this program to improve the 
education of children. We have demonstrated 
thabJ this program does make a difference to 
' children. We feel that we have an obligation 
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to the educational community to disseminate 
this information to other schools so that 
if they have resources to develop new pro- f 
grams 'they won^'t duplicate what we have 
already done. Number two: if we have an 
obligation to children for the betterment 
of their education, then we also have an 
obligation to dissieminate this information 
to other schools, Esseptially, there is' 
a moral^and ethical responsibility of edu- 
cators to share their information. 

At the LEA level, exactly what you describe 
- will take place. It will lie true the minute 

you receive JDRP approval. You will start 
to get requests for information, and you 
need to realize this in 'advance. There is 
: : , no economic benefit to the LEA to go through 
this- system. If you really believe in the 
^ project that you have taken through the JDRP 

process, then you have a certain responsi- 
bility to share this information. How far 
you share it will be controlled at the LEA 
level," 

12, How do you back out of th^ NDN if you get inundated 
with requests 'and your staff get over-committed 
along the lines of tWe previous question? 

Ahs: Very simply, you do not request funding .from 
. . the NDN, Only you can determine the extent 
of yourv dissemination practices. And these 
* ^ . V> are spelled out in the proposal j^op prepare 

for grant funds from the. NDN, • 

13, Is there an appeal procedure if your submission 
is not approved? 

■ ' ^. • ■ . ' • ' 

Ans: No formal appeals are necessary, as you can. 

come back as many times as'^you want with new \^ 



d&ta and a resubmission. 
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14* Dpes the p^nel use an objective ratirig scale, or 
does it rely on its impressions of the evidence 
presented? 

Ans: The latter. Panelists 'are ^^sking the 
' ^ questions: is the project effective, can 
^ it be replicated, does it help, students, 

teachers, etc.? The* panelists reach a judg- 
ment based on all the evidence presented. 
If thi^y approve a project, they are saying 
that it can be disseminated. It is up to 
the Individual LEA to decide to adopt it or 
adapt it to the local setting. 

.15. Does the JDRP expect you to document that .y(% 
program was the cause of the principal intended 
out^mes? » 

Ans: Yes. The panel requires that evidence be 
presented which indicates that it is t.ie 
project, rather than-other external factors, 
which is producing the beneficial outcomes.' 



16. Where can I go to get help in putting together 
tny, submission? ' 

Ans: The references lisied in Appendix C would 
^ . be very useful, especially -the JDRP Ideabook . 



\ 
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Appendix A 
JDRP Submission Format 



Appendix B 
Sample JDRP Submissions 

Developmental Career Guidance Project 



Pima County 
Tucson, Arizona 



Project Equality 
Highline Public Schools 
Seattle, Washington 



Broject -CAP 
Boston Mountains Educational Cooperative 
Greenland, Arkansas* / 




Appendix C 
References 
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Appendix A ^ 

' ■ • • ■ • ■ ■ . ' ^ • 

.- *FORMAT. FOR SUBMITTING MATERIAL ' 
TO THE JOINT DISSEMINATION REVIEW PANEL 

PROGRAM AREA: (e.g.. Title III, reading, career education, 

environmental education, education for the/handicapped) 



I. . INTERVENTION- TITLjE, LOCATION: ' 

Specify the title of the interven^tion and the 
location f6r which evidence , of effectiveness is 
being submitted. 

.II. DEVELOPED BY 

Indicate who developed the intervention orig- 
inally, even if this happened at a different site 
than the one foT wh^ch evidence of effectiveness 
is being presented. 

III. SOURCE AND LEVEL OF FUNDING: 

List all funding sources for the intervention at 
the location for Which evidence of effectiveness 
\ is presented; for each source, list the amount of 

funds. (See Figure I. for an example) . ^ 

IV. BRIEF DESCRIPTIOli OF«*^NTERVENTION: 

. Briefly describe the intexrventidn for which claims, 
of effectiveness are being made.. The description 
should coyer at least the following points: 

- What is tiie "intervention? 

- What are its objectives? 

— What claims of effectiveness are being made? 

- What is the context in which it operates? 

' . - Who are the 'intiended users and beneficiaries? 

/ , - What are the characteristics of the groups on 

* which tjtie intervention was developed and tested? 

- What are the salient features of the intervention? 

- What are the costs for adoption and maintenance of 
the intervention?^ (See Figure I. for explanation) 



*The submission paper must not excfeed ten' (10) single-spaced 
pages including any tables, graphs, etc. , 



4.6 



EVIDENCE OF EFFECTIVENESS . ' 

Describe the, evidence of effectlve|i6ss for the Interven- 
tion. This section should deal with each of the following 
points, although not necessarily in the same order: 

' ■ ' ' ■■; ■ ~ ■ 

InterpretaMlltv of measures; Evidence that the quantitative 
measures are reliable and valid Indicators of the effects 
claimed. . • 

■I , ■ . ■ ' 

Credibility of evidence: Who collected and analyzed the 
data, what assurances are there that the findings" are 
objective? j. 

\ Evidence of Impact: What is the evidence that something 

happened? What, are the effects clalriied for the intervention? 

Evidence of statistical rellabll-jtv of the effects : What 
Is the evidence that the effects happened often enough and 

with sufficient reliability to be likely to happen again . 
under similar circumstances? 

Evidence that the eff ects are educationally meaningful ^ 
Can alternative explanations such as practice effects, 

maturation, selection of superior treatment groups, etc ' 
be ruled but? °. r ' , •» 

Evidence of "generallzab lllty to the populations for w hirh ' 
@, the product or practic e Is Intended ;" Ev^HpnP>. m,^ 

-product or practice has been tested x^ldely enough and under 
sufficiently diverse circumstances to give assurances that ' 
the effects claimed may be similar when the product or . 
practice Is used elsewhere fori the populations Intended. 
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FIGURE I — 



An Illustration of a Table Shell for Showing Costs 



Source and Level of Funding of Intervention 

■ I 



Installation Subsequent Yesrs 
(Non-recurring (Recurring Costs) 
Costs ) . ' 



Personnel 

i 

Personnel Tr^lnir.:: 
'Facilities 

Equipment and Materials 
Constjmable 

Other. Costs* - Specify 



) 



Total' 



*e.g., transportation, technical assistance, ° public relations, etc. 
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The information reported herein was obtained pursuant to contract no, 300''7?''0303 
with the Office of Education: U^S. ^D^^partment.of Healthy Education^^' and7]^lfare, 
• Contractors undertaking issuch projects ^tmder ' mve^ment sponsorship are ^^enooiiraged 
to document information accprding to their' observation and' professional Judgment. 
Consequently, information^ points of view y or opinions stated do not necessarily 
represent official Office of Education ^ position or policy. ^ 



PROGRAM ARE/l: CAREER EDUCATION 



ACTIVITY TITLE, LOCATION : Project Equality, Highline Public Schools. Seattle, Washington 
DEVELOPED BY ; Highline Publ ic^chool s ^ . , ^ 



SOURCE AND LEVEL OF FUNDING : 1973-74 
, • 1974-75 
* • 1975-76 
1976-77 
. 1977-78 



BRIEF DESCRIPTION OF ACTI VITY: 



$35,740 ESEA Title III 

82,864 ESEA Title III 

71,268 ESEA Title III 

38,548 ESEA Title IVc 

36,800 WEEA 



The major thrust of Project Equality has been the reduction of sex-role stereo- 
typing in students at the K-6 grade levels through implementation of materials which 
counter such stereotypes in both occupatjafial and home sex roles. To accomplish this 
purpose, the sta^f of Project Equality h'ave developed three sets of materials, which 
may bis used independently or in, combination. 



Activity Description 

Each set o£ materials is designed to provide s^dents with nontraditiorial sex 
j^role models. The development of materials extended over the project's first three 
years of operation (1973-76) , 'With final field testing taking place during the 1977-78 
schooT year. The three sets of materials are:. 



!• Occupatiors^SjjDuWt ion Packets .. This set of material's cpnsists of five 
separate packets, each of which may be used alone, Two packets have been 
developed for use with grades K-2 students, two for students in grades 3-4, and 
one for students in grades '5-6. These packets are designed as a hands-on career 
education activity for elementary students based on the \isolated skill concept. 
This is the singling out of a saleable work skill requireid for a wide variety of 
jobs and already possessed, in some' measure, by the studertt. The student iden- 
tifies and uses the skill in a hands-on simulated work experience. As students 
apply their saleable skills in a variety of job-related activities, it becomes 
clear that: (1) a skill is not dependent on sex ; and (2) a skill required for 
one type of. work is often transferable to another type of work. Discussion ques- 
tions for teacher use with the class also emphasize these two points. * The 
packets include: ^ X 

* Color Discrimination - This packet uses the- skil 1 of colpr discrimination, 
' which is required in such- occupations as interior decorator and electrician. 
Activities include sorting bundles of wire -and socks by color. Time usages 

2- 3 hours over a 2-3 week period. Grade levels: k-2. 

0 Crawling and Squatting - This paket uses the skills o/Trawling and squat- 
ting and demonstrates how they are needed in occupations such as pVumber 
and stock clerk. Activities include stocking "g^ocery shelves" and crawling 
under "houses" to tighten pipes. Time usage 2h'3 hours over a 2-3 week 
period. Grade levels: K-2. 

© Assembling - This packet uses" the skill of assembling arj' item in a specific*-! 
sequence. This skill is required in such occupations as.. factory worker and < 
baker. Activities^ include assembling a bal Iplbint pen and a bicycle reflector. 
Time usage:. 2-4 hours oyer a 3-week period^. Grade levels: 3-4. \ 

0 Creativity - Xhis packet uses the skill of creativity, which is required in 
such occupation's as artist, weaver, and architect. Activities include ,, 
making puppets and weaving wall hangings. * Time.usage: 8-10 /hours over a 

3- 4 week period. Grade levels: 3-4. 4 



• Measuring - This packet uses measuring skills found in different occupations, 
including shoe salesperson, pastry cook, and carpenter. Special attention 
is paid to avoiding sex stereotyping in cooking aKd sewing occupations. , 
. ^ Activities include measuring a coat pattern and a wearer to determine 'if the 
coat would fit and measuring task accomplishment wi\th a stopwatch. Time 
• usage: 3-6 hours. . Grade levels: 5-6. 

Lessolj plans and many required materials are included with the packets. The , 
* ' materials show gr?at sensitivity to many kinds of discrimination: illustrations 
show a mix of rac^s as well as sexes, and a discussion of how to deal v/tth handi- 
capped children is included in the Color Discrimination ajid Crawling and Squatting 
packets. . \^ . ^ 

2. Yellow, Blueyiand Red Book . This packet contains a targe number of ideas for 

, short-term activities which help the teacher and students^to expand their ^are-* 
; ness of sex-role stereotyping, and -broaden their view of sex roles in tfie home^ 
• and OOF appropriate' job opportunities available for all qualified people, Time-^^|R\ 
N usage: 3-5 hours over a 3-week period. Grade levels:' 1-6. 

3, Many Thousand Words - Work Pictures . This looseleaf book contains a set of 

- 8 x10, b\ack and white pictures of women .and men , and girls and boys in a variety 
; of non-stereotyped work settings at home, school, and in the community depicting 
various skills and abilitijes. Discussion questions focus on the skills , , 
needed and on whether pei^ons of;either sex might hive those skills. Time 

usage: 5 hours over a 3-week period. Grade levels: ,1-6. ' ^ 

-\ 

' CoiTOon Characteristics of Materials . All Project Equality materials^ have been 

designed to have the following characteristics: 

" • ■ " ■ * ' ■ , ■ ■ . ' • ■ 

. e They provide content material which can expand' students* perceptions of 
occupations open to females and to males. I \ 

0 They fit within the context of subjects the teacher is already expected to 
cover in the classroom. ^ " . ^ 

. 0 They are self-contained. " 

0 They are easily adapted to different classroom settings. 

e They do not require any additional teaching or support staff members>for 
implementation. 

Measurement of Achievement . The Project Equality staff has also adapted the 
"Who Should" test, ^ from work originally done by Dr. Lynne Iglitzin of the University -^v 
of Washington.. This* test is used with all materials and attempts to melsure changes 
in students' perceptions of occupational and home sex roles. It wiVl be described'in 
detail in the "Evidence of Effecti vteness" section of this submission^ . 
^ ■ ■ ' . V. 

Goals and Objectives , . 

The goal of 'Project Equality is to expand students ' perception J of occupations 
>open to fema'les and those open to males. The following objectives were developed to 
meet this "goal : ' ^ . ^ 

Objective 1 . Elementary students in grades K-6, where any one of five Occuoation 
Simulation Packets is used appropriately, will have expanded their perceptions 
of job options open to females and male§. By the conclusion pf the treatment, 
period, these students will score significantly higher than a control groxip-Of 
students drawn from the same grade levels on the "Who Should" te5t. 

Objective '2 . ElementarS' students in grades . 1-6, where the Yellow, Blue, & Red^ 
Book is.xised appropriately, will have expanded ttjeir perceptions df job options/ 



open to females and males. By the conclusion of the treatment period; these' 
students will score significantly higher than a control group of students drawn 
•from the same grade T^els on the "Who Should'^ test. * , 



Objective 3 . Elementary students irf grades\l-6, where Many Thousand Words - Work 
Pictures are used appropriately, wil 1 have e^a.nded their perceptions of job 
options open to females and males. By the coiKlusion of the treatment period, 
thesn students will score significantly higher' than a control group of students ". 
' drawn from the sam^ grade levels on the "Who Should" test* 

In these objectives, th.e term "students" means those boys and girls Lho have both 
participated for the entire treatment period and also have taken both the pre- arid 
.posttests. Appropriate usage means usage for at least the number of hours suggested 
earlier for each packet or set of materials. " • 

* . • . ' . 

Claims of Effectiveness 

All three sets of materials were fie?d tested-during the '1977-78' school year i'n - 
two school districts in thq Seattle area. The .objectives listed. above were mef in a 
majority of cases, leading to the conclusion,jJ^^^^ students have expanded th^r occu- ^ 
patlonal perceptions to include non-stereoty|jp^cl jobs for both sexes?. r 

Context V ' jl ■ 

- Background ; Project' Equal.ity's home sijte.y/as' the Hi^^^ \ ^ 

Washington. The impetus for the projetl^^ study of 258 district fifth graders 
in 19^71. The study demonstrated tha't botfvv^^^^^ and>^irls tended to See jobs in terms 
of masculine and feminine stereo.typesv^^^ of the fifth / 

grade girls fell heavily into feminini^-'Ster^0^^^ Staff of Project Equality •. ^ 

s(2t out to provide materials which w#?^; fr&e' of sex^stereotyping and which demonstrated 
to students tha^t people of both sexes &uld perfi3hr*m|i^^^^^ of job tasks- 

Field test materials . Project ma terj a Is. were dey^l,^^^^^ 
the first three years of project operation. Final : field testing in 1977-78 was conducted 
outside the Highl ine District to assure that no student exposure to materials had taken 
place in earlier years i The results discClssed in this submission are from two districts,. * 
the Northshore and Bellevue ^^Sehool Districts, also near Seattle. These two/districts, 
as well as Highline School District, can be characterized as predominantly whit.e 
middle-class, with 90% or higher vyhite populations. - Northshore School District covers 
an area whfch is rapidly evolv,ing from a farm economy to an industrial/residential 
area. The* Bellevue School District contains. mainly middle and upper middle class 
families, with a high percentage (67%) of professional, technical, and rMnagerial 
employment. Per-pupil expenditures in the Northshore School District were $1,386 in 
1976r77, .whi^e in the Bellevue School District they were $1,587. > 

Intended Users and beneficiaries ' . ' 

The intended users are K-6 students of bptlrvsexes and various socioeconomic and v 
cultural backgrounds.. The activities are simple,^ int6y5esting, and expefijential , 
making them usable with students of ^^lifferent ability .levels. ; - ^ 

Characteristics of Students Involved \ , ' 

\ ^ Like the total populations of the school districts, the K-6 students with whom 
these materials have been used are primarily*white and middle class. Achievement 
levels vary, however: latent figures from the annual statewide assessment of fourth 
grade achievement, usipg the Comprehensive Test of Basic Skills battery, show Norths ' 
shore students sco^rtng at the 58th percentile on the average, 'while Bellevue scof^ed 
at the 69^11 percentile on the average. ' (Students. in the Hlghl ine District, where 
materials were originally dfevelpped, scored at the 48th per^n tile. ) Each student . 
[Jopulation was comprised of approximately eq,ual numbers of boys and girls. 
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Salient Features 

. The most importSnt features of these m'aterials ^are summed up by the followinq" 
point?: , 

• The materials present sin .le , interesting, 'bias-free-activities which demon- 
strate that work skills can be possessed by persons of both sexes. . 

9 They are self-contained and can either be. used individually or in, combination 
over the K-6 grade range to provide students with a longer series of non- 
stereotyped activities and concepts. 

0 The materials are relatively low .in cost a-nd the program can easily be imple- 
mented within any K-6 curriculum. . \ j . ' 

Costs . J , , ' • . 

■ . ( . 

. Presently; Project Equality staff members -train local district staffs to imple- 
ment materials. Ifis suggested that, adopti ng districts plan to involve the follow- 
ing .personnel: the district career education director and/or durriculum director; 
an elementary principal; an elementary school librarian; and six elementary teachers 
A distnctXadopting all Project; Equality materials should pl-an to assume costs for 
•the followirv"- 



'J ■ ■ . • 

A 14-day training session for staff numbers implementing the materials. 
Two i$"-day;followup meetings after the training session 



'0 'Adoption of Occupatioa Simulation Packets 
6., Adoption. of tfie YeTlow, Blue, .and Red Book 

0 Adoption* of the Manv ThniJ<;anfl WnrH<: - Wnrlc f 



Adoption of the Many Thousand Words - Work Pictures { - * • , . ^ 

,0 Evaluator for field testing^Qrf materials (if desiredjl^ 



, : . ^ Table 1 

ApprbxMrriate Costs for Adoption of Project Equality Ma^teriarls 

' staff .Development - , Rlnt Year Implementation Costs 

^ / ^ District' Implementation Team: ^ ' ^ . ■ $650.00 " * f 

' , 1 Career Education Director ' - . ' 

.1 Elementary Principal * ^ - . 

4^Ele!Tientary Librarian Substitute ' • - 

. 6 Elementary Teachers at $35/day ' ' 

Ma te rials 

Occupation,. Simulation Packets: ^ ^ 

K/2 Crawling and Squatting * 275.00 - 

K/2 Color Discrimination . 200.00 ^ 

3/4 Assembling , ■ 180.00 

. ; • 3/4 Creativity 300.00 • ' 

5/6 Measuring 365.00 
Yellow, Blue, and Red' Book 50. OO*""" 

Many Thousand Works -Work Pictures 50.00 * . 

Contracted Services \ ' ' 

Evaluator for pilot testing of materials 200.00 
.2. days at SlOO/day 

. *■ ' . . ' . ^ i $2270.00 

The estimated^costs for operation in subsequentVyears (replacement of 
lost or damaged items) are $75. OO! ^ 

Per pupil costs . Since these materials^ould be used by all district 
elementary schools in turn, pier pupil costs woul-d depend on total number, of 
district K-6 students. Assuming a district had 3.000 students „ per • 
pupil costs would be:^ • / 

First yearf- $0.76 

Subsequent yeJrs: 0.03 ^ V 



• EViDEkCE OF EFFECTIVENESS ^ • ' . 

Iriterpretabil itv of.. Measures ' ^ ' / " 

The "1*10 Should" test was the outcome measure for a]l sets of materials.' There are 
two forms of this instrument: /one for grade levels K-2, and the, other for levels 3-^6.' 

K-2 form. The K-2 form has 26 questions which are read to students by the 
teacher. The students have only the answer sheet, not the questions. Two sample 
questions are: ; "At home, who should wash dishes? Men? Women? Or both?" "Who can . 
use a hammer better? Boys? Girls? Or both the same?" All questions are an?wered by- 
the students by circling a picture of a boy, or of a girl, or of a boy and girl stind- 
..ing together. - ^ . 

. 3-6 form. The grades 3-6. form of the test has four parts with aSota?;! of 41" 
Items. As in the K-2 form, teachers read the questions aloud to students. Part I • . 
ij^*^,,^^ ^"^^ carpenter and nurse; students circle "man," "woman," or ' 

both to indicate who should do the job: , Part. II lists six jobs done in class such ' 
as eraser cleaner;" response options are the same as for Part I. Parts III and IV 
list jobs done in the home. Response alternatives to Part III are "fathe^," "mother," 
or - both," and response alternatives toj^art IV are "men," "women," or "both." - . 

Reliability^and validity. Experts /Ks^x-role stereotyping at the elementary 
. 1 evel revi ewed the tast^and- conf i rmed /hei r^orften t val i di ty . Test-retest, reliability 
and Item validity were assessed in a school district that had not been ex^sed to 
instrument- and materials development activities apd where ^Somen's rights were not 
fot-mally Included anywhere in the curriculum. ^ 

^ 'vThe test was administered twice to the same students two weeks apart. Host 
materials required^bou.t two weeks o/ usa^e, 50 two weeks corresponded to the inter- 
val between pre- and posttests for both treatment and control- classes ill this study 
: The test-retest^qrrelations were .62 for the K-2 ,fdrm (administere.cLto 20 first gr^de • 
studeRts) and -B8 for the 3-6 form, (administered' to 22 fourth grade students) . Item 
validity.was-^^enimfed by computing biserial correlations betweerr each' item and„the 
whole test scoPer^he coefficients for the K-2 form ranged from .19 to ..6Q with an ■ 
average of .52: For the 3-6 form they ranged from .41 to .73 with an average of °.59. 
Minor modifications to the test were made on the basis of these data: 

Credibility of Evidence 

■ ' • ■ ■ " ■ ■ ^ ■' ■ ■■ 

For the 1977-78 evaluation of student outcomes, all tests were administered by 
the students' classroom teachers^ All teachers had received two days dtf-.training .^^^ 
followup consultation from the Project Equality.staff onhow to implement? the materials 
and how to conduct the pre- and posttesting. During these sessions, teachers were' 
instructed to read test directions /^nd items slowly, to* stay als close to the script 
as possible^in order to keep inst/uctions similar for all classrooms, and to use a 
neutra^l tone of^ voice. Teachers /Were told tb repeat questions, if necessary, to 
alloW^ students enough time to tMnk over the guesti'ons before marking their answers. 

The K-2 form was administered in two sittings during the same day>/^The 3-6 form 
was- administered. in one sitting. : Scoring was done by- project staff, with two persons, 
.indapenldentl-y scocing each test as a check on accuracy. Scores were4<ey punched for 
computer analysts and all key punching was verified. Project staff members were al^o 
responsible fq^ final analysis and interpretation of resylts. " 

jce of Impact 



Evidejc 



Evaluation design . The study to- assess impact was conducted with K-6 students 
in the'North$hore and Bell evue School Districts. The study was designed to provide- 
pre- and posttest measures of performance, on, tr^eatment^nd control students for : 
- eaclt'packet or set. of materials. The rationale, for thiV design assumed that real 



student' growth could be assumed to have occurred if the group mean pretest difference 
was jTot -statistically significant and group mean posttest difference was statistically 
significant following use of any one set of materials with the treatment group. 

- - Jn each district, treatment schools were selected from the middle range of average 
school-wide achievement scores within the district, Fqr each treatment school , 
another district school was selected whose average achievement scores matched those of 
the treatment school as closely as. possible. Because of 'practical, constraints o/i 
evaluation in actual school settings, teachers in the ^selected treatment schools were 
asked to volunteer to use Project Equality materials'in their classrooms. District 
field test coordinators selected teacher participants from among these volunteers, 
paying particular attention' to -achieving a mix. of teachers which was representat-ive 
of the whole district. Control school teachers were also volunteers and were selected 
on. the same basis. The total number of teachers implementing Project Equality 
materials in the two districts was 37, while the number of control group teachers was. 47. 

Data analysis. The same, outcome measure was used regardless of which matenials 
were taught; . therefore, the control students' scores at each grade level were combined 
into one group, whereas the treatment students' score;s were separated according to the 
materials taught. This accounts for the much larger 'N's for the control than for the 
treatment groups. This was judged to pose no problem in^anal/jzing the data, since 
variances/of treatment and control groups were quite similar. . 

Only those students who took both> the pretest and the posttest were/included in * 
the analyses. . All students in a given grade level who were instructed in the same ' 
materials were combined and then compared with all -control . students at that grade 
level. For each cortipaVi son i a \t-test of indefJgndent means was computed on pretest 
means of treatment and coptrpl/groups and a separate t^-test was computed on posttest 
nfeans. Separate .analyses were performed for, each Occupa'tion Simulation Packet , for 
the Yellovy.-, Blue, and Red ;Bodk, ahd for -th^ Mairy Thousan-d Words - Work Pi ctures , ^ 

ResuUs , Data tables fbr^each analysis are separately presented and explained 
below. .The topypf each table corisist-s of a bar grapVof^the-mean.gain of each group ' 
at each grade, leyel , together with a table of the means and standard deviations on 
the pre- and px:)5ttest. . The .bqttom of-each table shows the value of the t-test and 
the corresporidlngisignificance level for each comparison . ' ' — 

Occupatiba Simulation Packet: .Color Discrimination ^(Table 2) . The treatment 
.Snd control groups started at almost the same point, as shown in Table 2. The 
treatment group significantly out-performed the control group on the posttest. . • 



Table. 2 

Color Qiscrimination (26 items) 



Grade 
2 



.Mean Gains 



Pretest 



Posttest 





Gain « 5.20 


Mean 


SO 


Mean 


SO 


T 




12.39 


4.35 




17.59 


4.66 


C 


i Gain » . . 18 . \^ 


12.32 


4.-25 




12.50 


'4.29 


] 


12 13. 14 15 16 17 18 











T » Treatment Group C ' Control Group 



■ - ■ i t-tests ■ . ' 

[) Pretest Posttest 

Gfrade . J_ t_^- df sl£ ,.t df si£ 

2 41 156 '.09 195 NS &.32 195 < .0005 
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Crawling and Squatting (Table 3) , The first 
' the " 



Occupation Simulation Packet: ^ 

grade treatment group scored close to the control group on tHe pretest but significantly 
above it on the posttest. In the setond grade, the pretest difference was signffi- ' 
caatly in favor of the xontroT .grouR; yet the Targe gain ir\ treatment group scores 
makes the posttest difference significant in favor of the treatment group. At the. 
kindergarten level , the treatment group showed gre,ater gains than the control group, 
but/these were not significant. r r 



- Tab']e 3 \ 
/ Crawling and Squatting (26 items) 



Pretest 



Posttest 



Grade 




Mean Gains * 


Mean 


SO 


Mean 


SO 


K 


T 


. . Gain . 2.05 


15.05 






17.10 ' 


' 4.81 


• / 


C 


Gain « I.64 ^^BHB ^ 


14.80 


.5.62 




16.44 


. 5.52 - 


1 


T 


Gain . 4^53 l\VV\VV\V\\\i , 


. 13,07 


3.94 




17.60 


4.18 




C 


Gain • .24 > 


12.66 


4.90 




12 -.90 


4.88 


2 


T 


KV\\VV\VV\N\WV\I .Gain - 6.63 


'8.55, 


3.58 




15.18 


3.83 




C 


\ 1 Gain ■ .18 . * ' 


12.32 


4.2S^ 




1Z.50 


4.29 



9 10 11 12 
.T * 'Treatment Group 



13 14 . 15 16 , ^ ir- 
C ■ Control .Group 



18 



N 

Grade , X*" X. 
K 20 , 41' 

1 30 166 

2 11 156. 



t'tests ^ 
Pretest Posttest 
1 slq t df slq 

.15 59 ■ NS: .48 59 NS 

■ .Sq 194 NS 5.52 194 :<.'00q5 
'3.33 ,165 <. 005 2.22 165 <.025 



: Occupation Simulation f^acket: Creativity (Table 4) . The creativity activity 
was highly effective at both grades 3' and 4, its shown in Table^4. At neither grade 
were the pretest scores significantl/ different for treatment and control groups; 
however, following the treatment irrterventioh^ the treatment group far out-performed 
the .contrbl group at^ both^^^f^de l^eTs, • • . . ' • 



Grade 



T 



-X Table 4 

) Creativity (41 items) 

Mean Gains , Pretest Posttest 

Gain » li:80 • ^>ean SO 

LV V^\V V\V\\V\\V\\\\VV\VN 18.04 7.33 

■■■I Gain > 1.74 17.01 7.04 



kV WV\ VV \\\\VV\V-V^ Gain . 9. 
■■i ^Aln " 1.35 



10 



17.38 
18.68 



5.92 
6.42 



17 18 19 20 21. 22 23 24 25 26 .27 28 29 30 
T . ■ i jtjatment Group C ■ Control Group 



Grade 
3 - 
4 



49 
21 



196 
177 



Pretest 

.89 ■ 2^3 
.90 196 



t-tests 



ill 

;NS 

NS 



^ 1 

7.87 
3.82 



Posttest 

»di. 

243. 
196 



ERLC 



57 



ftean 


So 


29.73 
18.75.. 


8.82 
J. 38 


26.4a 
20.02 


?7.03 
9.38 




> 


<:0005 
<.0O05' 






i 




T . .• • 



■ \ 



Occupation Simulation Packet:> Assembling (Table 5) , This activity was highly 
effective at the third grade but not effective, at the ^ fourth grade, as shown in 
'Table 5. v 



Grade 



. Table 5 
Assembling (41 items) 

Mean Gains 
Oaln ■ 11, 80 — 

Gain • K74 



Pretest 
Kean SO/ 



?osttest 



18.70 
17.01 



.90 



E3 Gain • . 

BB Gain • 1.35 

16. 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 
T" Treatinent Group C " Control. Group 



21.73 
18.68 



9.^ 
7.04 



7.51 
8.42 



: t-tests. 

. N Pretest Posttest 

Grade X X 1 df sig df < sig 

3 , 20 196 .81 214 NS' 7.86 214 <.0005 

4 48 177" , 2.43 223 <.05 1.76 223, <. 05 



I Mean 


SO 


* 30.50 


6.13 


18.75 


8.38 


22.63 


9.00 


20..02 


9.38 







^ Occupation Simulation Packet: Measuring (Table 6> . This activity worked well 
^"e fifth grade, as shown in Table 6. While the sixth grade treatment group gained 
more than the control, the difference was inconclusive due to the significant differ-^ 
ence in pretest scores. ' • . 

J 



Grade 



. • , Table 6 
Measuring C41 items) 

^ ^ Mean Gains 

Gain ■ 8.45 LV\\\KAvV^\!^V Vv^\^V\\N 
Gain • 2.96 "^ 



Gain .3.81 
Gain • 1.86 



Pretest 


Posttest 


Kean 


SO 


Mean 


' io 


21.82 


8.05 




35,29. 


7.89 


1?.15 


8.75 




22.11 


10.02 


26.26 


9.32 




30.07. 


10.22 


21.35 


8.81 




23.21 


10.62 



19 ^0 21 22 23 24. 25 26 27 28 29 30 
T ■ Treatment Group C • Control Group , 



Grade T C 



17 194 
'27 115' 



t- tests 

Pretest Posttest 
t df sig t df sig 

1.30. 209 MS ' 3.96' 209 <,0005 

2.49/ 140 <.'05 3.12 140 <.005 



Yellow. Blue, aifid Red Book (Table 7) . At all five grade levels tested, the- 
treatment group achieved significantly higher posttest scores than the control as 
shoWn in Table 7. At grade 3, the pretest score difference was also significant; 
however, the posttest score difference was much larger. ' 
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Table 7 



Vellow, Bl ue, and Red Book {26 items at Grade 1, 41 items at grades 3-6) 



12 



' 14 , 16 18 20 
T ■ Treatinent Group 



Grade 

1 ^ 
C 


^ ^ , ■ Pretest Posttest 
Mean fialns ■ Kean SD ^ Mean SD 


k\\VV\\V\\] Gain • 7.33 " 
■ ^jain ■ .^4 ^ 12.66 


6.50 
4.90 


21.00 
12.90 


4.86 
4.88 


3 ^ 
C 


Gain - 7.00 kXWVWVVM 

Gain s 1.74 HIHI 1 7 m 


9.33 
7.04 


29.58 
18.75 


9.39 
8.38 


4 ^ 
C 


Gain > 12.79 k^{VVV\VVV\V\xVV\V la ng 
•Gain » 1.-35 i^H 13.68 


6.83 
8..42 


30.88 
20.02 


9.29' 
9.38 


5 

C 


^1n . 6/70 kV\\\VS^\\J - r ^n.7n 
Oiln >-2.96i^^H 29.25 


• 7.99- 
- 8.75 


,27.40 
22.11. 


'12.90 
10.52 


6 ^ 
C 


G4ln - 6.75 |\X\\\\^V\1 ?n aq 
Gain ' ^•86 jm 21.35 


10.20 
8.31 


27.64 
23.21 


J,0.71 
10.62 



Grade 
1 
3 
4. 

: 5 

6 



N 



22 24 26 28 30 
C » Control Group 

t-tests 



32 



6 
26 
33' 
20 
47 



166 
196 
177 
194 
115 



, t 
1^38 

^94 
.44 
.82 

. .27 



Pretest 

df ,s1a. 

170 NS 
220 <.01 
208 NS 
212 NS 
160 NS 



/-t 

4.01 

5.34 

6.15 

1.78 

2.40 



Posttest 
""If siq 

170 <.0005. 

220 <. 0005 

208 <.0005 

212 •<.05 

160 <.01 



Many Thousa nd Words - Work .Pictures" (Table 8) . As shown in Table 8, the treatment 
group outperrormed the contror group at grades.!, 3, 5, and 6. At grades 3 and 6, the 
.pretest difference was-also significant, but the-gain. in treatment group performance 
was larger." Growth at the fourth grade level is approximately equal ^for the two groups 



• ' . ' >. ' Table 8 

Many Thousand Words - Work Pictures (26 items at grade 1. 41 items at grades 3-6) 



. Pretest 



1 ' 

c 


[Zi:^:^ GaV- 2p^^, • i3.87 
. ■-Gain^^.:«ife - _..>r^ ^ ■ - - 22.66 


.'3;6.9' 

4.90^ 


16.74 


4.01 
^..88 


3 

Cr 


fc- ■ /: 'py~Jyy?y/\ ^aain:^ 5;]^. 21.04 

■■flU Gain ■ ■1.74' ■ 27,01 


" 7.66 
7.04 


"26:^0 
i8.75 


8.42 
/8.38 


4 ^ 
C 


■• " ; jr77\ Gain -1.63 ■ • 18.26 
■1. Gain - 1.35 18.68 


6:46 
8.42 


19^ 
?€L03 


7.34 . 
9.38 


5 T. 
c 


y//// / //A fiain . 19.52 
^^^■^ Gain - 2.96 ' 19.15 


7.68/ 
8.75 


' 25.74 
22.11 


10.19 
10.02 


6 ' 

c 


Gain - 6.95 V/////////\ 26.48 
Gain > 1.86 21.35 


11.16 
8.81 


33.-43 
23.21 


8.^4 
10.62 



12 



22 24 26 - 28 30 it 34 



Treatinent Group 


C -Control Group 






- Grade 

. ■ 1 


N 

iT Su 
38 166 


' ■ '■ \ ■ . t- 
\ " Pretest 

\ i* . ^ 111 

• V 1.71 202 NS 


tests 
5.10 


Posttest'. . 
df jfi£ 

202 <.0005 


■ 3 


25 196 


2.50 219 <.01 


4.17 


-219 0005 


'4 


19 177 


, .26 rV^^ : NS 


.08 


194 NS ' 


5 


^73 194 


• . 34 265 NS 


2.61 


265 <.01 


6 


23 115 


2.08 136 <.01 


5.06 


136 <.005 
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Educational Significance -of Resul ts 

In order to better display the amount of student growth after' implementation of 
Project Equality materials. Table 9 has been constructed to--5hQW score gains^for the 
treatment students in standard deviation terms. Each value in rable 9 represents 
the treatment students' score gain divided by the pre- or posttest standard deviation, 
whichever is larger. That is, each valjje is equal ^o: \ . . 

/ posttest X - pretest X \ 
_ ■ S^larger ^ 

The larger stajodard deviation was chosen so that- the result would be the more con- 
servative estimate of student gains. 

As shown, six of the seven materials produced gains of over one standard devia-- 
tion at one or more grade levels. ' Eighteen out of 20 gains were greater than one- 
third of a standard deviation- Gains of one-third standard deviation or more v/ere 
obtained at every grade level that was tested. These results were obtained in the 
"relatively short time of two to three weeks, using only one set or packet of materials. 
It is probable that use of more than one set of materials over a longer period would 
increase impact futher.v . - 



Ratio of Treatment Students' 


.r Table 9 
Gains to Larger 


Standard Deviation (Pre- 


or 


PoStt^Sty 








Grades 














2 


3 


4 


5. 


6 




' Yellow,. Red, & Blue Book 


1.13. , 




.71 


" 1.38 


.52 


.63 




Many Thousand Words ' 


. . .72 




.62 


.22 


.-.61' 


.63 




Color Discrimination 


,. r ■ ■■■■ 1 


.12 












.. . Crawling & SqOattIng 


, .34- " 1. 08 I 














Creativity 






1.33^ 


1.29 








Assembling 






1.31 


.10 








• Measuring 










1.05 


.37 ■ 





General izability v. - ' * : 

• The evidence given above demonstrates that these materials are highly effective • 
with populations Such as those in* the Northshore and Bellevue School Districts. These 
populations are mainly white ai>d middle class. It is worth noting that all materials, 
especial ly^those for direct use with students, are unusual ly attractive and j)resent 
* simple, intriguing activities which make-;U«tf?ff skills most students already have 
regardless af their achievement levels. ' ; ' . ^ 

Concl usion ^ * ^ _ " ^ ' . 

Washington State law.and national Title IX legislation both speak to decreasing 
sex discrimination in all aspects of the education system. In recent studies it has 
been found that schools, -for tl^e most part,- are not fully demonstrating the past, 
\present, .or future participation of females in a variety of workv^roles in textbooks , 
audio-visua4 materials , career eduta tion programs . and teaching strategies. Across the 
country educators are* attempting to develop and acquire media" and materials vyhich 
presjent- -females and males ifi non-stere6typed occupational and personal roles. The 
mateVials outlined in these objectives fit this requirement, and have been tested in 
an experimental atmosphere and found to be effective. Project Equality's contributions 
are a positive step in the direction of improved student sel f-concept, which can serve 
to benefit all students ahd the educational system in which they function. 
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PROGRAM AREA : Career Education 
— ^ 



ACTIVITY T ITLE. LOCATION : Boston Mountains Educational Cooperative for Federal 

Programs— Career Awareness 'Program (CAP) * ■.-.p 

Greenland, Arkansas 

DEVELOPED BY: Project CAP staff, Boston Mountains Educational -Cooperative 

SOURCE AND LEVEL OF FUNDING : Funds for the first three years of the project 

- ^- (1974-1977) we're provided by an ESEA Title MI 
' Grant and by local school district- funds . 

I Grant Period - L&vel of Funding 

1974-75 Title*III - 71",062 

* Title III 52.885 Local 17,628 

- : 1976-77 Title III 32.624 Local 30,228 

, 5156,571 ; $47,856 ' 
. Total = $204,427 

YEARS OF ACTIVITY^ DEVELOPMENT : 1974-1977 

* ' ■ ■ ■ 
BRIEF DESCRIPTION OF ACTIVITY 

Goal . 

Projett CAP students, grades 1-8, will be significantly more aware of careers 
than a similar student population not involved in the project. ■ ' ' ^' 

Context arid students served ' 

, ■ ■ — ~ ■■ "S * . ' 

• The Boston Mountains Educational Cooperative. is a consortium of eight-school 
districts in a rural, inountainous area of northwest Arkansas. ' The cooperative pools 
the resources of several school districts to develop needed programs for area school 
children, programs which might not be available to individual school districts 
Project CAP serv.es students from six of the' consortium districts, five of which 

are located in Washi ngton County and one of which is located in Madison County The 
estimated population in the geographic area served is 16,000 persons -With a school 
enrollment of 5,100. Ninety percent of the pot3ulati<)n is from Kura If arm families; 
the remaining 10% are considered rural non-farming families. Fifteen percent of 
all families in Washington County and 33.1% in Madison County are classified as 
having^an income less than the poverty level. The schogl population is predominantly 
Caucasian. . . . 

Claims of effectiveness 

: The claims of effectiveness are that Project CAP staff-developed materials and 
related educational activities result in significantly increased career awareness^- 
among elementary and junior high students from low-income, rural areas. . „ 

Program descripti 



: CAP'-curricufum .' The curriculum was developed to. show students that school 
subjects are important and related_t6 the world of work by pairing a specific 
academic skil^with a selected occupation.^ For example, the skill- of telling timg 
on a clock is linked to the occupation of cosmetologist in a learning packet that" 
des;:ribes the importance of keeping track of time so customers are- not kept waiting.- 
The curriculum has been revised to include a- scope and sequence chart that presents 
occupations and career education conce[>ts to be. taught across the 15 USOE clusters 



as well as a self-awareness cluster. For each grade level,* two packets in each 

of the 15 occupational clusters plus two packets on self-awareness were developed 

for a total of 32 packets at each level. The occupations selected were varied 
in regard to .entry-level requirements. 

' CAP materials . The materr^fs include cluster units (with accompanying bulletin 
board materials) , teacher di rections', student packeTs, and testing instruments. 
Packet lists are distributed to participating .teachers so they can request CAP ■ 
- packets to fit into their- lesson plans. Teachers are encouraged to use packet^ at ' 
the rate of at least one per week. 



CAP packets . -Packets targeted for use in grades 1-6 were "designed to be 
completed in 15 or more minutes of class time and to be infused into the ongoing 
curnculum. Packets for grades 7-8 were designed to occupy the same amount of 
class time, but they are occasionally used as separate units rather than being 
infused into the curriculum. The packets contain a career script that presents 
required tools, tasks, training, work conditions, and economics as well as the 
concept of work as a way of life. Emphasis is given to developing student aware- 
ness of personal preferences and to participation in decision making, with the 
4Dpportunity to be qreative in problem solving. The packet teaches an" academic 
skill by demonstrating how- a worker needs thait skill and by. providing. practice 
^^^ly^^^f^;-, Posttest in each packet covers both career 'concepts and the .academic 
skill. Bulletin board materials of printed letters and pictures, as well as a * 
suggested layout, are provided. Teachers are encouraged to vary or^add to the 
materials in any way they choose. 

- ■. - t 

Project staff.; The CAP staff consists of a project director, two curriculum 
coordinators, a deputy evaluator, a bookkeeper-secretary, and aJmafcHine operator. 
One consortium district (Greenland) provides office space for the project director 
and for the coordinators. Each coordinator is responsible for thVee schools. The 
coordlnators'^o to the-.schools one day per week so that there are weekly meetings ' 
with teachftrs in each school.* The coordinator's meet with elementary ^ind middle 
school teachers as well as principals >and superintendents. The de^ty evaluator 
visits each school once per week, while the. project director makes periodic 
visits*, - : j . '\ ' _ ^ , ; ^ 

Staff development . During inservice days 'at ^the start "%f each school year, 
coordinators meet with teachers^^ and principals to present samples of new curriculum 
units (packets) for the benefit af both new and returning teachers. During the 
year, training is' cofiducted on an indiyidualized basis during" the weekly coordinator 
visits. Each teacher receives trainfng;jConcerning how to use and vary the instruc- 
tional packets, , while the coordinators, receive^ Suggestions for improving packets 
and ideas for developing new opes. V ' ■ ■ ... 

Weekly status report . A short report is completed eacb'^w^ek- by every parti- 
cipating teacher and principal, providing a communication ehanhel-with project ' t 
staff as well as a- double check, for -^CAP administrative channels./ ' . 

Pa rent- community involvemen t. A r^jor concern of the project is parent- ' 
community involvement. Newsle^ers containing articlies . written by teachers and 
students who are participating in the program are sent home with- each child. 
Teachers are given a Resource People List that indicates community members who 
wil'l visit clashes, or. whom, groups of students may vis it^ to discuss otheir occupa- 
tions. Parent^ are invited to class to describe their careers. The project director 
and curriculum coordinators meet with the P. T. A. and school board^oups to discuss 
the CAP program. / • • 

Costs ■ ' ; 

The 1974-77 budget for this project was $204,427. The six participating 
districts paid $47,856 of that amount. The average cost per student per year 
was $22.56. Costs for salaries included' the project director, two curriculum 



! 



coordinators, an evaluator, a bpokeeper-secretary, and a machine operator. The 
school' districts provided office space for the dfrettor and coordinators as well 
as space for packet development work, printing, and packet storage. The following 
table presents costs for the. three years of project developrfent- ' 



Table 1 : Costs 





1974-75 


1975-76 


1976-77 


Project Personnel^^ 


^ -'59,382 


58,061 


55,328 


School Personnel n^ining 


3,993 


2,966 


3,741 


Equipment, Materials, and 


7,687 r 


9,486 


3,783 


Consumables 


571,062 


$70,513 


$62,852 



The estimated costs of replicating the Project CAP program in an adopting 
school are shown below: ^ 

Table 2: Repl ication Costs 

~~Tr EM , . Cost 

^iet of 32 consumable packets [ $1.00 per student* ' 

Teacher's manual for each grade level $2.00 per manual* 
Consultant fee (optional) . $200.00 (2 days of a Project CAP 

• . consultant; at $100 per day) plus 

4 ' travel expenses 

*does not include shipping charges ^ 



EVIDENCE OF EFFECTIVENESS . . ' ' . 

— - ■ ^ - ■ « ■ >i * 

Interpretabili ty of Measures • 

Project^CAP staff administered two data collection instruments, one project- 
developed and one developed by an independent^: test development organization. 

The Career Awaren&ss Test (CAT ) . The CAT is a staff-developed instrument 
comprised of 30 multiple choice items and designed to measure knowledge of 
selected occupations and the training/education required to enter them. The 
instrument has three different versions: one for grades 1-3, one for grades 4-6, 
and one for grades 7-8. It is designed for group administration and the questions 
are read orally by the examiner. ^ 

Validity of the CAT . Prior to the drafting of items for the CAT, a revfew 
of relevant literature and materials was conducted. The Encyclopedia of Careers 
and Vocational Guidance , the Occupational Outlook Handbook , materials from Ohio 
State University* s Center for Vocational Education, from the Oklahpma State Depart- 
ment of Vocational Education, from Cobb County, Georgia, and selected commercially 
published materials were reviewed. Based on this review. Project CAP staff selected 
15 occupations from each of 15 career clusters.* These occup^atibns were selected 
on the. basis of present and forecasted demand, and were representative of the full 
range of positions on the. career ladder from entry level through profess^ional leveT. 

*Rroject CAP staff utilized the U.S. Office of Education's classification of 
occupations into 15 clusters. . . 



Using the total of 225 occupatiohs across the- 15 career clusters as their 
reference group. Project CAP staff wrote a minimum of 10 test'items for each of 
I^VJa V ^° 1?°'"® ^'^^'"^ lli'Clusters were admini- ' 

in ?ti^n!iE •° ""^'^ ^^ V":^ ^^J^^ ^° ^^■"'^ most discriminating items. 

An Item analysis was conducted on the results obtatned from each of the pilot 

5n that; J 1^° Items from-^ach cluster that best discriminated students according . 
to their knowledge of occupations werp combined to form separate 30-item versions of 
the instrunent for each of the three levels. " version^ or 

R eliability of the CAT . To, determine the reliability of the three versions of 
the CAT. eacn version was administered to 100 students at the respective grade levels. 
Each version of the instrument was divided into two equal parts and the scores on 
.one half were corR^lated with the scores on the other half by computing Pearson 
S^n Spearman-Brown l^rophecy Formula). 

The following internal consistency reliability coefficients-were obtained^ ' . 

Leve l r 

* Grades 1-3 .68 

' Grades 4-6 - .77 

Grades 7-8 ' .71 , . - ^ 

... The Career Knowledc^e Test. 'Published by Evaluative-Research Associates. Inc., 

Louis, Missouri, the Career Knowledge Test is designed for students in grades 
III ^""^^f l^^'Z knowledge of the world of work. Project CAP staff selected 
the Career Knowledge Test because the items included in it test knowledge of 
occupations, which are representative of all 15 USOE career clusters. It- is a 
30-item picture instrument in which items test knowledge of the similarities and 
differences of various occupations and' of their required tools. The reliability ■ 
of the instrument is shown by a reported internal consistency reliability coeffi- 

The Orient ation to Career Concejjts/Worker Activities Scale . . Also published 
by Evaluative Research Associates, the Orientation to Career Concepts instrument 
.includes a series of scales designed \to measure career awareness of students in 
grades 4-8. -Project'CAP staff selected the Worker Activities Scale because it ' 
measures knowledge of what different workers do in various occupations that are 
representative of the 15 USOE career clusters. It is a 20-item scale; items are' 
written in a five-response, multiple choice format.' The reliability of the scale 
IS evidenced by a. reported internal consistency reltability coefficient of .76. 

Credibility of Evidence 

~ [ : — : ^. 1 • \ . ■. 

Evaluation design. Project CAP staff used a "posttest only, matched control 
group model to evaluate the effectiveness of the project activities To se?ve as ' ^ 

pTte7 n'J r rntp%'''-5''°S'S^'" Mountains Cooperativl' that ar c?- 

pated in the project, si )^.schools outside the Coop having no formal career educa- 
tion programs were selected. These schools were matched with the project schools • 

KLh? *i ^t^^?^'^^'^^ attendance; (e) average assessed valuation; (f) geo- 
. tex?I;if va°"ab?«.^^^ socioeconomic factors; (h) curriculum; and (i pother con- . 
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Evtdence of cSroparabi 1 i ty -of treatment a n d contro.l schools . 

^ ^ — ' — ^ * '■ ' — ' 

• Population^^^tneafment and controrschodls 



7 



School f 



Project Schools; Populati 



1 i. 



El kins 
Farmington 
Greenland 
Prairie Grove 
St. Paul 
West Fork 

TOTAL 




8 Total 



34 34 30 43 49 22 44 ^0 306 

54 41 56 49 51 53 59 58 421 

23 27 32 29. 38 45 45 47 2'86 

88 76 71 80 67 71 56 58 567 

28 .27 44 : 21 28 30 20 • 35 233 

65- 63 56 54 70 77 67 88 540 

29'2 268 289 276 303 298 291 336 2353 



Control 

School 


Schools: Population 

' ' \ . 2 3 4 


Grade 
5 6 


7 


8 


r ] 
Total 


■ Charleston 


• 47 53 


41. 53 


42 51 


•62 


71 


420 


Deer ' ' 


29 18 


23 27 


26 23 


23 


25 


194 


Eureka Spri;ngs • , 


32 38 


38 ^3 


30 28 


36 


34 


279 


Green Forest 


48 58 


51 54 


. 68 72 


•78 


66 


495 


Pea Ridge 


53 37 


37 34 


56 " 43 


51 


42 


353 ' 


Yellville . .J 


' 55 41 


54 54 


45 55 


50 


57 


411 


■ TOTAL . 


* 264 245 244 265 Z6J 111 


300 


295 


2152 



• the average number of teachers at the six treatment school's was 33.3; the 
average number of teachers at the'six xontrc^l schools-was 30.3. 



f Expenditure/A'DA' of treatment and control schools 



' Treatment Schools 
El kins ~. 



Farmington 
Greenland 
Prairie Grove 
St. Paul ' 
West Fork 



Expenditure 
Per ADA 
$795 ' 
^ 723. 
838 
755 
884 
697 



Control Schools 
Charleston T 
Deer 

Eureka Springs 
Green Forest 
Pea Ridge ij 
Yellville 



Expenditure 

Per .ADA 
~J712 
764 
1,013 
756 
. 719 " 
736- 



t Average assessed valuation, ADA, and expenditure/ADA 



6 Treatment Schools 
6 ContrO'l Schools. 



Average 
Assessed 
: Valuation 

$3,583,285 
$3,798,705 



Average km 

604/ 
581 ■' 



Average 
Expenditure 
Per ADA 

$732.00 
783.50 



• The six treatment and six control schools are geographically located in 
"the rural , mountainous area of northwest Arkansas. 
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Socioeconorpic factors: the occupations of parents, from the -treatment and 
control .school districts incl ude J imited farming, poultry raising, cattle 
raisiKg*' truck^and orchard crops, logging, l/jght industry, and service jobs. 

Jhe^ curriculum of all twelve, schools appears comparable, e.g., 9 of ^e 
twelve schools use the same reading series. ' . * 

^ The student populations from-all twilve schooTs appear to have comparable '- 
.exposure_to media outside the school. All areas have access to a weekly 
, . countypaper and in' a few of- the locations a daily ri,ews Jape r. All areas 

can receive at least one clear network TV channel ?nd an , educational 9ha"rtnel . 

^ - Data collection -procedures. SttiJen£s in the treatmetvfe- and control scht^ls were 
administered the UAl, the Career Knowledge Test, 'and the Orientation .to Car^r 
Concepts/Worker Acti-vitiegJ Scale in'late February and March 1977. The instruments 
were, administered by project CAP staff after they had received speciatj^ninq in ' 
? administering them, including practice in reading the questions orallyHc reffne- 
their pronunciation, pacing, and manner of administration. . Project stafiKscored 
al student responses to the instruments. Tjne project evaluator -was r^sponsibte for 
all data ana ly-ses. " / ' 

Evidence of impact " . c > 

. . Data analyses. Mean posttest differences between treatment. and control schools 
by grade leveT were analyzed using the t-test for iridependent samples. Tables 1 2' 
and 3 below present the means, standard deviations,- t-values, ahd "significance levels 
tor treatment and control schools on the Career Awareness Test (€AT), the Career 
Knowledge Test, and the Orientation to Career Coacepts/Worker 'Activities Scale The 
results presentedjin these* tables demonstrate. that the treatment schools ijivariably. 
out-performed the control schools by a wide margin on the posttests. On all three 
instruments, at every grade level, group differences -were hfghly significant 



Vr 



Tadle 3* 
f Career Awa rune's s Test .(CAT) 



Grade Leve+- 


N 


Mean 


• : SD .. . 


t- value " , 


1: Treatment 
1: Control 


292 
264 ' 


. UA 
; 8.3 


2.7 
2.2 


19.79** i . 


2: Treatment 
2: Control 


268 
- '245 


.17.3. 
12.9 


.4; 3 
3.1, / 


13.44** 


3: treatment • 
3: Control 


28,9- ' . 
244' 


19.0- 
15.0 


3.9 , 
•3.6 


12.28** ' — 


4: Treatment • - 
' 4:^ Control ^ 


^ ■ 
276 ' 

^ 265 , 


14.8 
'13.0 


. ;;3.4 ' 

3.1 


J 

B.O'^** * 


5:' Treatment^ 
, 5: Control 


303 


16.9 ' 
14.4 


3.9 

3.2 * 


■ . 8.16** 


6: ' Treatment 
6: Control 


298 
272 


17.1 . 
15vl , 


..3.8 
, • 3.3 


6.64** 


7f Treatment 
.7: Control' 


291 

300 ' ■ 


■ 15.5 
13.4 


r.3-' 

3.9 


6.64** 


8:^ Treatment 
8: Control ' , 


337 
29^ - 


17.6 
14.6 


.10.1 

3.''8 


■4.84** 



**s1gnif leant at the .001 Jevel 



.Career Knowledge Test 



'Srai 



etsJLeveT Mean SD - t.- value 



n 


Mean 


SD 


292 


20.9 ■ 


4.0 


264 ■ 


' 16.5 


4.5 


268 


20.5 . 


. 4.3 


245,' 


... ,.18.6 - 


- 4.3 


289. 


22.4 


■■■''■3.8 


244 




3.6 



1: Treatment ■ , . 

1: Corkrol ^ .264 " ' 16.5 4.5 ^^-^^ 



2: Treajrtifent — . ^ ...y • t.. . , pp** 

2: Control 245..' . ,18.6- -4.3 ^-^^ ^ * 



3: Treatment — . .... 

3: Control ^ * . 244 , 20.8- , 3.6 * 

**significant at the .001 level ^ - '. . , 

' ' ' ; ■ ^ . '■ ' ■ . , 

> ' Table 5 ■ • ' ^ 

OrientaticfTrto Career Concepts : ' Worker Activities Scale 



Grade Lev^ei • N Mean ' SD t-valiie 



Mean 


' SD 


9.B • 


3.0 


» 8.r ; - 


2.9 


, /. . 




■lo-.'s.^-' - 


- "3.O.V 


9:2: " 

■ ■ 


■ 2.9> 


11.4 •• 


• 3-.-r 




3.3' 


^ ■ 12'.8: •' 


- 3,3 


r .12.1 


,2.8 


T3..5 . 


5.7 


.'12.5 


3.1 



/ 4: Treatment , ' ' ' 276 . ^.^ ^.u ■ c 

4: Control = - - 265 " - S.l :; - 2.9 * 

5: Treatment 303 -\ -i^.^,. - ^ r 

; ■ 5: Control - 267 ' 9:2; " - 2.9> ■ ■^■^'^ , 

6:,. Treatment " >^298= .11.4 ' 3/1 -7 nA** 

6: 'Control V" ^ 272/: ' ' a.d / - V ? ^' - 7.30 * 

> ' 7: Ireatment. ' 29lV- ..... . .... , 

' ' .7: Contror' > • ' .300 .12.1 - ^ ' 2.8 ; ."V^^ 

■ , • v. , . 

■^•8:: Treatment " ' 337,". r..... . „^ 

8: Control. " 295, .'12.5 3.1 >: ■' 

; • *f signi/,iaant at the .QOl l.evel f * /. ' . 

" *signifi^ant at the ^005 levelV ^ , ' -.fv " ^ 

Evidence that the effects are educationally, meaningful ' . 

. TKe e.val ua-t ton plan used, the data collected and the resulting an^l-yses/ ' , 
provide reasonably convincing evidencevtf^t^'Pr^^^^ CAP prod uce'd educafionaHy 
si^gnifican?: results.' Thfe treatnient and .cQnt'CQl schools were equivalent 'on such 
' variables as^chqol population, 'schqo.1 staff ^pte'r student,^ district expen(i44:;ureT . : 
per ADA, averjige^ dai ly' a'ttJ^danc'e, a>^rage. a^ssessed valuation, geographic l;ocation, 
^ socioecqnomic * faptOfSCf ;CU|riGijli^^^^ afnd. exposure to informStiort' outside the*^ school . - 
.Th% use of compa^abl€ cdW'ol schools 'app;^ans^/td rule out the passibifity' of explann 
ing the results in ternte of other variables such as maturation effe^^ts, practice , 
effects, and outside-^school/ experiences. In additiofi', all mean differences between 
; treatment and control schools exceeded.one-J^hird of /a standard dey^iatfon in- fact, ■ 
for grades 1-3 the treatment^. m6an ^scores exceeded, the control mean scores by ^r*^e 



,than one sfandard devvation on the Career Awareness Test. The largtf sample that 
included grades 1-^8 arhd the very high level, of statistical, significance attained 
across all grades indicate that the Project GAP goal of significantly iacreasing 
students' awareness of careers wias achieved/ \ , . \ ' h : 

Evidence of general izabil.ity to other "populations / * > 

V . The high degree of stringency with which the ev^lu^tion was /implemented and 
the exceptionally strpng »s:esults provide reasonable assurance that ^e program 
IS making substantiar progress toward meeting the cari&QLr awarene^ 'needs of local 
istudents. Those school districts with similar rural populations^omprised pre- 
dominantly of Caucasiaft students may consider adapting the Project GAP. program 
to their local, settings, knowing that the program has been demonstrated to be 
effective in its original locale. 
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^ PROGRAM AREA ; CAREER EDUCATION 



■ A'CTIV-ITY TITUE LOCATION : ^Developmental Career Guidance Project, Plriia County, 
• •' ' ' Arizona ' . 

DEVELOPED BY : Pima County, Arizona , Devel opmental Career 'Guidance Project * ' 

SOURCE 'AND LEVEL OP FUNDING : 1971-72 

V . 1972-73 



1973- 74 

1974- 75 

T 



$278,000 Arizona State : Department of Education 

500,000 Arizona State Department of Education 

535,000 Arizona State Department of Education 

560,135 Arizona State Department of Education 

10,000 Educational Professions Development Act 



BRIEF DESCRIPTION OF, ACTIVITY 

"The intervention is a set of activities across levels K-12, designed to help 
students develop krrawl edges and skills in the fol lowing areas : self-awareness and 
self-esteem; the world 9f work;. and decision-making. 

Goals ^ . ^ 

■ The goals o/th4 PimacCourityrDevel opmental Career Guidance Project are defined 
by the Arizona Careerv^ducation Ma^^ developed; and approved by: the Arizona State 
Departmept' of^ EducatioVvoth the he>D of local district and school staffs. The 
matrix includes goal statements that ^re. divided into seven categories. Thjese' cate-^ 4 
gorles are listed below along wit|i example goal statements* taken from four grade 
level groups (K-3 primary, 4-6 intermediate, 7-9 junior high, and 10-12 high school): 

0 S e 1 f - Awa re n e s s ( j u n i 0 r h i g h level): The students will consider their intere^sts 
and aptitu^s in exploring career information. \^ 

-o ^Caree^ Awareness (primary level): \. The studen^ts will become aware that people ' 
; ■ " . : \do different things at their work. ■ 

0 Decision Making (intermediate level): The students will recognize the steps 
of the decision-making process. , ■ 

' © Employability Skills (high school level)::^The students will present an 

accurate description of education, tr^aihing, experience3 and information 
about themselves to potential employers through avvarietv of means such 
^ as intervi^s, tests, and application forms. ■ . ; 

0, Educational Awareness (junior high lev^l):, The students will understand the 
educational requirements needed for entry into occupations within selected 
career areas. ^ . ■ . ' ^ \ 

0 Economic Awareness (primary level):. IHie students wri 11 become farhiliar w'^th 
. the varied economic rewards gained from different .kinds of work', ^ - 

0 Appreciations and/Attitudes (high school level ): The 'students, based on their 
tentative career choices, will analyze the interrelationship of work, 
continued learning, the arts, and-Meisure in achieving soc^l ijesponsibjllty 
and self-satisfaction, ' ^ 



Some^of the seven categories are stresj^ed more at one level than at others/ 
In^grades K-6, the, building of sejf-awareness> self-esteje^^ in decision- 

making IS viewed as paramount, though awareness of cfareer areas is not ignored.^ 
At the 7-9 levefl , . occupc^tional .clv^ers are studied in more depth . Students are/ 
encouraged to examine their Interests , continue developing their decision-mating 
skillSi'and spend time thinking about and examining/several potential career, areas^ 
At^thj^lO-12 VeveU direct contact with selected career areas is provided,; including 



-exposure to work, to workers,,; ahd to^actual work experience. ' ' 

, - ^Stffden.t^^^^ is influeneed not only by what educators do, but 

also by what^^jg^, employer^ community groups do. . Therefore*, the goals of 
P4m Countx^ increasi^ng the involvement of these groups in 

career educafion-qnd imprWFfig^the^ in whatever roles 

they may, pi ay. ' . . > ■ - -^^^4-^-. / ' ^ 

Claims of Effectiveness V v. ^ "^^^ ■'\ • ' V " > • f 

As a result of high levels of exposure to this: cll^Seg^^ucat^ program, a 
sample. of students in grades 4-12 in the county performed bettec-Jn all categories 
tested, as measured by a locally developed Careef^s Test ,- than didV'ictmp^tr.able ^ ^ 
sample of students with Tow exposure to the program. The evidence pr^sSh ted to . 
support this claim is from the' school year 1974-75. . ' ^ 

Descrip^tion of Intervention Activities • ' - 

. The approach to career education in Pima County is often referredrto as infusion 
Infusion is not 5o much a change in the content of school subjects as "in focus and 
intent. It involves the continual demonstration of the many reVationships between, 
school subjects and particular occupations, job clusters, or th6 world of work as a 
whole. For example, addition may b^e taught using restaurant checks in a simulated 
coffee shop instead of using blank paper. Because 6f tffe nature of infusion, and 
because of the wide array of career education resources that the Pima Cqunty project 
has made available for school staffs to choose among, the specific experiences that 
make up the program for any one student vary. A few examples may help the reader 
understand. the nature of the program, but it should be remembered that these are 
representative examples, not universal student experiences. 

Elementary level activities . ^Elementary level activities general ly .focus on 
iself-awareness, self-esteem, and an introduction to jobs. 

Example : In one elementary sch6ol , a Care Center was instituted as a 
resource for all K-6 students. / Records and record-players, books, games, 
bulletin board materials, and (Jther activities were available to help 
students learn about the two Themes of the Month, one concerning an 
occupational area and thfr other concerning an affectfve area. Example . 
areas examined in one month were careers in the transportatiw cldster 
and problems in deal ing witlx crisis si tuations (death, divorce; illness), . -y 
. -vStudents used tTje Center both for class assignments and also independently 
to investigate careers and explore or express feejing^. The Center staff, 
parent aides, or the school counselor were available at^air times to 
work with students; also, students were encourag'ed^o expre^ _^ 
feelings^' a Feelings' Box^ vyhere they could ;in^t notes telling their 
\ ■ feelings ah(^ asking to talk to a Staff member if they wished. The focus . 
on each month's occupational cluster culminated in a full-day Career Day, 

: ' with several speakers coming in to discuss their careers. 

■ I - • ^ ■ ^ _ ■ ■ ■■ • . 

Junior high level activities . Activities at the 7-9 level focus on greater 
study^f occupations anjd application of decision-making skills. 

\ Example : \ In one 8th grade claS^s, a* unit on occupations required students 
^, - to complete a notebook consisting of values clarification, decisioh-makinq, 
and job exploration. exercises. Examples of exercises were: (1) a values-^ 
^appraisal exercise in which students rated (on a 1 to 10 scale) activities 
• of importance to" them individually; (2) a job interest exercise where 
students checked which of 10 activities under various job clusters they 
would rather perform; (3) an Occupational Study Guide asking students 
specific questions concerning the job{s) of their choice; (4) a data- 



. people-things' exercise where students selected a response representing 
Qne of these preferences in 20 situations; (5) a jobs , decision making 
exercise where students Visted 10 careers of . interest 'together with 
• charactjeri sties of each career (education required, wheth^r/it involved 
otheT<peop1e or not, risk factpr, preferred size of organizatioRi, and 
^other factors of the student's ^choice) ; (6V a What I J,ike About Me exer- 
cise;where.:students cho^-wM attributes described them; 

and (7) Va. Where . I Hah^- To Be Ladder: where students forecasted* which of 
their desires in life they might fulfill in five years or in their lifetimes. 

Senior high level activities . Activities^ at the 10-12 levels are aimed at 
giving students actua^ exposure to work and work sites. \ < 

Example ;. A communications laboratory of one high school was the center 
of radip/TV/f ilm/newspaper activities' on campus. Students learned basic . 
technical skills in one or more of these areas and then produced their; 
own films,> broadcasts, and newspaper. This involved covering campus 
and community activities, ^interviewing people on and off campus , visiting 
local TV and radio $^t^^^^ and newspapers to observe and participate in 
a real setting, arid prMaring original material (such as a videotaped 
spoof of television advertising). 

These examples portray Targe-scq^le efforts which have been undertaken during 
the Pima County effort. Hbweve»i, smaller connections made day-to-day between : \ 
sc|ro^. and work are even more conSfTon. The example given earlier of doing- addition ' 

m-restaurant checks rather than blank paper is illustrative. 

' . ♦ /...•■ 

Project staffing . Career educat-ten in the county^ is coordinated-^^jsi-facil ir 
tated by the Pima County Developmental Career Guidance Project. Pr<)ject policies 
are set by an 1.1-person governing Board consisting of one official from each of* 
11 participating school districts, operating under an interjurisdictional agreement,.' 
The staff consists of four teams of persons working out of separate offices serving . 
different county areas. The central office in .Tucsoac\is run by the prpject dir^ector 
and staffed by guidance special ist^ andv suppoY^t staff . The threeNarea' of f ice5^^^ 
each located in a small district, outside of TucsoVy, are each headed by an'assi€larit 
director and staffed- by guidance special tsts plus|i small support staff. '* 

Guidance spMfal ists .' The-^'f^^^ line" persons the Pima County effort as 
implemented in .1 974-75 were the project's guidance specialists. Persons chosen as 
guitlance, specialists generally had advanced degrees in counseling pi us^ commitment 
^to and enthusiasm for career erjcaiion. In 1974-75" they were usually assigned 
three to fQjdf .-s^^^^ .isit\ng^ a weekly basis. Numbering ^abdut 30 x 

in :^that:Schodiiyea^ guidatice spetialists^were chiefly responsible for helping 
te,achers and counselors infusfe career education and career guidance activities 
into the classroom. Through workshops, in-service sessions, and individual consul- 
tation they strove to impart to teachers and counselors an understanding of Pima 
County goals of career education and the nature of the infusion process; to acquaint 
them- with career education media and methods for their use; to help them plan the 
use of guest speakers, iField trips, and bt^er^community . resources; and to help ^them 
integrate guidance activities into the classroom. Often the guidance specialists 
conducted tjfese activities themselves, particularly at the elementary level ; in 
fact, the degree of j)roject-staff invol vement directly in the classroom when compared 
to other infusion approaches to career education across the country .^is . a significant 
feature of this project: ^ 

Staff relationships . Copperation between the project staff and staffs of the 11 
participating^ districts is another notable feature or this program. Districts offer 
support to the project in many ways, such as supplying space- for offices and work- 
sh^s and sharing salary costs of some staff members.. In addition, the Board 



distributes' project funds in part on ^e basis of district needs instead of on a 
strict per-pupil basis, allowing small^* districts to maintain resources comparable- 
to those maintained' by larger dtstricts\ : 

^- Cooperation among grotfl)s involved in^ this project extends "also to contacts 
between project staff and building staff. Project services are offered throughout, 
the county, coordinated by the 'central, off ice and three area offices. However, no 
school or teacher" is required to use prpject services, and in fact project staff 
will not begin wo^k in a. school unless both the principal and a majority of the 
iteachers reque^t-it. Use of project services has spread from six schools initially 
/in. 1971-72 'ti3i virtually all county schpdls in 1977-78. This appears to be because 
students, parents, teachers, and:;administrators have learned ;of the project's • 
intriguing activities and impressive results in other schools and have asked for 
projectl^^ta^ff involvement in their schools. - • 

Community Resource Center . One ofj^he project 'Ns mJijor activities is its ' 
Community Resource Center, located at tffe central XfC.son o/tice. The three ful 1 - 

'time Center staff members coordinate a wi\te^ range, o]f contacts betweer> the schools ■ 
and th^cdmmuni ty, .in^Tuding^'^g^^ vol unteers,. summer businessman/ 

teacher seminars, and Vork expdsure/experieilce sites, for studei^s. The Center's 
CommuhitV Resource^ Advisory Counci 1 consists\f 18 community members wtjo work with 
Center staff members to increase community invo^^vement in' career\ducaftlont The^ 

'effort has been airjied 'at fncluding parents, community organiz^itions, and all 

"vsegments- of the world of wpfjk as col 1 ab'orators in education, who bring their own 
unique and varied contribution's and ^vjewpointsTnot meT&^ly supplementers^ who do 

• what ^educators tell them to do. Business and community group"membe^ act as< speakers , 
resource persons, and work exposure/experience sfte hosts, under overall coordi/iation 
by the Community Resource. Center and its Advisory CoidnciK In aclditTon, these ( 
people participate in seminars and work exposure activities for teachers and colin- 
selors as part of the latter's in-servke treKiwhig'. . 

•Involvement of parents is a particular focus of the project': throi^h the Com- 
munity Resource Center. Experiences offered at the ^.Center include discussion 
groups' on career education and on effective parentino, leadership training, and 
opportunities to participate in school activities ds teacher aides , speakers, or 
resourxe persons. Project staff members are ful ly cognizant of the fact that paren- 
tal e^^pectations influence! students' career-selections substantial ly. . Staff mem~berf 
work to i-ncrease participation in* the on-going parerflFgV6ii|^^^ J 

Career education media .' Another njajor project re soufte is it/* extensive collect' 
tion of career education media,, located at. all. four off ices ^and a^iUble to all 
county teachers and counselors . The .prbj^t constantly Maintains an<J updates in ^ * /. 
extensive library of career education .media 'and materials for use by all Pima' County 
teachers, including commercially publ is]ied material s 'arid 24 project-developed instruc- 
tional units. Its Media Advisory Commi ttee, made up of teachers^, counselors ,.^and 

project staff members, representing different county *5treas , careful ly screens all 

incoming materials for effectiveness and abs&nce of race or se^x bias; the Committ^^^^^^ 
recommendations are valuable for local school purch6si ng decisions as vielT as pfoje^ 
decisions. • These material s , speakers, and activity units have been available for ~ 
a1 1, local scho^)l personnel to select from according to the'' needs of their students.; 

Inservice training . Inservice program's for teachers and counselors consist of' 
regular one-hour after-schpoT sessions conducted by guidance-special ists , held -at - - 
least once a month. . Special sessions to help teachers'and* counselors develop or 
tailor materials to their students' needs are he! d whenever requested. -In addition, 
special topic workshops are held several times a year-cDu/ity-wide pr for individual 
districts, coordinated by-the project's two staff development specialists. . Teachers 
and counselors are also encouraged to take relevant col lege course work; - 




Context ^ ' 

: Pima County. 1s a highly diversified area, containTiqa one of Arizona'^ major 
cities (Tucson) and also large stretches of farrfi land and^-lmlian reservation land. 
Its population is about 80% white and.^20% Mexican-AmericanV^Amerlcan Indian, and 
other ethnic minorities. In general, whites tend to live ihtne urban portions 
and minorities tend to live in the more rural areas. • ^ 

The county contains a total of 155 schools: 104 elementary schools, 34 junior 
high schools, and 17 senior high schools. The total number of students In the 
county rs about 93,000.^ 

Intended Users and Beneficiaries ' ' . . , 

The primary users an^benefictarieis of the program are the K-12 istudents them- 
selves. Prdject staff .believe that students will benefit from the program, by being 
pnablied to better plan their school course work and activities around areas of ^ 
career interest; to bet^ter develop post-high school plans best ^suited to their ^ 
career aspirations; and, in the long run, to choose more appropriate careers and 
lead more satisfying lives than students without such ejcperiences. \ 

to accrue to th^ school and coinmunlty • 
oup members) who bejcome more f-amlTiar 
their r^l atfonshlpsyto each other and 



Subsidiary benefits are also believed 
(Including parents, teachers^ and communi 
with each other's roles and more effective 
to K-12 students. ■ . v - ^ : 

Student Characteristics - f . 




h"ma County project fprmed a cross- 
The student population-Ws 



The stiR^nts in schools served by the 
section of K-J^ students county-w.ide. The student populati^oTr-Ws approximately 
20% mir\or1ty, primarily Mexican-American btJt also American Indian, Black, and other 
minorities. The students represented all socioeconomic levels -and came from settings 
ranging from heai^il^ urbanized (though, no portion of Tucson could be characterized as 
'Mnner-city'* in l^e^usija^sensej to ejctre^ , > 

j 5al1ent Featur^ — . 

The central features of this ^effort are: " . 

e its emphasiis on refocusing the county's entire approach to education to 
d^tfstrate the relevance of education' and work. To achieve this end, 
1. __^_....va._.__-_ _ . • • 3 wide-^array ''of career education 



^rge numbers of activities." are* used 
resources for all; levels i^. offered; 

its use of spe'cially trained guidah 
career ^edu€«^ti on resources and implel 



md 
md 

speciaTis^s wh^ 
lentatione^^thods 



can keep abrea 



St of^ 



Costs 



The costs of ^d(31pt\ 



operation in 1974-75. The breakdown of costs 



this program can be estimated from the costs of project 



for 1974-75 is' as ^follows : 



rsonnel : , $403,924 

rsonjiel training n 2,000 

Applies and eqL^^fent 35,307 

onsumables v3,200 
ther '(contractual servicfes, ] 

travel, miscellaneous) 11/5,704 

Total. • $506,135 
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In 1974-75, about 75,0 were served fc)y project staff, resulting in a 

per pupil cost estimate of about $6.75. 

EVIDENCE OF EFFECTIVENESS • . . ; . 

• ^ ^ . ~ . ' •■■ ■■ 

Evaluatbr . 

' ■ ■ i ■ ■ ■ • ■ ■ ■ ■ [ ■ ' ..' ' ' ' 

Bej:)av1oral Research Associates *of Tucson, Arizona, developed the student 
outcome instrument during 1975-73 and. conducted formal evaluations 'of the program 
in 'subsequent years/ 

Interp'retabil i ty of Measures 

'~ • . . • ' ^ * 

Behavioral Research As-sociates developed, pilot tested, and field tested a 
"Careers, Jest" . to measure effectiveness of the program. In 1974-75 , the test had ^ 
two forms, an elementary-intermediate form for students in^arades 4-7 and a secon- 
dary form for stude/its at the 8-12 levels. In addition to.questions seeking infor- 
mation about students' background and home environment, tWe 4-7 level test contained 
49 items, while the 8-12 level test ^had 82 items. 

; Both forms! of the Careers test were paper-and-pencil instruments'. Both covered 
the seven categories of the Arizona Career Education matrix already discussed 
(except for the'^mplcvyability skills category at' the elementary level ) . In addition 
to scales in each of these areas, two 'other scales were included: a measure of 
cumulative knowledge in nine career! clusters, and (at the second/iry level ) a measure 
of interest in those nine flusters. All category scores consist of percent of items 
answered correctly out of all items in tnat scale, so that-a perfect s^ore equals 100 

Vajlidity and Reliability 

. . . • • • ' •• • 

Content validity has been verified by frequent review by state* county, project, 

and local district staff. Reliability of the test was estimated by correlation of 

test/retfest scores of 100 students tested at eacFof the two levels with a two-to-r/' 
three month separation between tests. Scores in th6^ seven categories were cbrre- * 

lated .70 to .90 from one admi nitration to th|. next^ - 

• ^ • . - : • .\ . . • - / 

Credibility of Evidence r " ^ * 

Behavior^ Research Associates conducte^^two to three ho.lirs of training with 
each giiidanceiyspec-ralisti to make-sure that test administration was standardized. 
^ill testing wasr conducted. by the -guida'nce special ists .o Data, were checked, analyzed, 
and interpreted by the evaluators." \ 

Ev idence 5f Iippgict ' \ . 

Eyaffati on design' . The .l974n.7s\valuat1on utilized^ posttest only, treatment/ • 
control group design, .^he 1974-V5 school year presented the last opportunity to 
find enough schools in the county to create a plausible Qontrol group. /After that 
school yea^ ngdst schools in the county were receiving project servicesj <rln February 
and early Karch of 1975, the guidance specialists used information froTi{ their weekly 
logs fo, assess^jbhe degree of Implementation of career education in all district ■ ^ 
schools. Bas^d.onthis information,^ schools were .divided into high, medium, and 
low implementation groups. ' A strictly no-impletnentation group was precluded because 
constant teacher and^ principal tran'sfers .among county schools had left 'almost no 
^^career edut3?ation-free'' schools. The 26 schools in the high^ implementation group ^ 
contained the treatjjent population anid the 25 schools,in the.low implementation . 
^group contained the cqntrol population. Appro^cimately TO . patent of the county's 
swiools received services tailored to non-Enql ish-^peaking^vS^udents who were 
-heavily concentrated in rufal areas\ It should be hoted that these school s>vtfere 

^ ; •. •.• ^' ^. . •• ^- ■ • • • ^ ■ \ / \- 

. • V • ■ ■ / '0 



not included in this. study in either the treatment or the control groiip 'because 
of. their special character. - . : 

Sample selection . Student sampling vTas done by random selection of intact 
classes, A <list of all teachers* names from th^ high and low inplementation 
schools was compiled. Student samples were constructed by randomly selecting 
fifty teachers' names inXeach group and designating their students as members of 
the -treatment and control groups.<j^ Twenty'-^f i ve\^teachers each were selected at. the 
4-7 level and the same number was selected at tt^e 8-12 level. The total number of 
high-exposure students actually tested at the 4-7 level was about 550, with an 
equal numbei;' of low-exposure students tested; almost 700 high-expoSure and 700 
lovy-exposure students were 'tested at ^the 8-12 level. 



the 



The numbers of schools, teachers, and students actually participatinq in 
evaluation are summarized in Table.l . 

* Table 1 

Numbers of Schools, Teachers, and Stu^dents Participating^ 
• in the Evaluation ! ^ 



/r 





No. of SchooT^ 


No. of Classes 


Tgtar 


Approximate 
Mb. of Students 


Total 




Grades 4-7 


Grades 3-12 


Grades 4-7 


Grades 3-12" 




Grades 4-7 | Grades 3-12 




High Implemen- 
tation Tii^oup 


, 17 


9 


22 


25. 




, 550 . . 700- 


1,250 


Low tmplemen- 
tatlon Group 

Total 


17 


8 ■ 


23 ' 


25 


\j8 


'550 700 
nibO- ]\.400 


1^250 


/ 

/ t 


17 


45 


50 


^5 


2,500 • 




> Comparability of samples . Tables 2 and 3 contain ,the results of^nalyses that 
were conducted to estimate the degree of-comparability of-the high and low exposure 
groups on dimensions other than amount of services received from^the Pima'County'. * 
program staff. . ■ ' ' - . ■■ 

" ■ ■ ^ - '.t-. ■ . 

c' • ' . Table 2 / \ . ■ ' . 



Characteristics of Teachers' in Sampled Classes 







Average Teacfifer 
Age 


. /. ■. 

Average' Education 


Average Teaching- 
Experience 


Ethnicity 


Sex • 




High Implementatioh 
• Group (N • 47) 


33.9 yrs. 


B.A. ♦ 28.7 units 


9. 7. years 


. 30^ • 
minority 


41 feniale 
6 mal^^^^ 




\ow-4mp1ementat1on 

Group (N -'aey 


■ 34.2 y/s. ' 


8. A. * 29.1 units 


9.8 years 


^ /27* ■ 
minori ty 


41 "^le ' 



■ /.. 



'Table 3 





Average School Reading * 
S^cores for Grade 4:*" 


Average Suchool Listeniiig. 
Scores fo]^ Grade 4*^;; 


High Irnplementatfion(N=16) 
Low Implementation (N=17) 


' ' ^ 2.9 — . 
3.1 





♦Stanford Achievement: Test administered to third graders in 1973-74. Scores' v 
reported in Grade Eqifivalents using publisher's norms. ' ^ 
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These data suggest thcit the high and low exposure classes do not differ .. • 
systematically on any^ducational dimensions likely to cause differences on the / 
Careers Test. . ' ' ' , , 

' I ^ ■ i . 

Data analyses ) T-tests were computed at eacTi level (4-7 and 8-12) on each 
career education category of the Careers' Test. Some ca'tegories were pada up of ' ' 
sub-categorjes; in l^hese cases, comparisons were done on the sub-categories instead 
of the tota/1 categories. Results are shown in Table 4. Calculated t-values are 
shown togerther with their significance' levels (to .001 mininum) Student scores • 
shown represent mean percentage of items correct on that categdVy or sub-category. ' 
All comparisons, at every grade level favored the high-exposure students; 19 of' 20 
comparisons favored the high-exposure students at the, .001 level of significance, ■ 
with the. remaining comparison favoring -them at the .02 level of 'Significance. y 

The sel f-awareriess category is not measured directly. by the test. However', 
.items 1ji the 1974-75 test did measure student perceptions of: (1) their readiness 
for employment in each of the nine -cl usters; (2) self-perceived brightness in • 
relationship to other students; (3) self-expectatitDns. for scholastic performance;- 
and (4) differences in certainty of achieving edutational and occupational aspira- 
tions. While these, self-report data are- rather '^oft" in character, it is note- ' 
worthy that en every comparison made at each 1 evel , .high-exposure student^ rated 
tWemsel'v^ higher than did T|)w-ex[josure students, indicating a higher degree of ' ' ■ 
self-conndence.„ ' ' 

Evidence of Statistical Reliability of Effects ' ♦ ' i '/ ■ ' \ - 

Data in Tables 2 and, 3 demonstrate that neither teadher- effects nor pre- . 
existing differences in the student gr(^ps are 1 i kely 'qxpl^nation'^ of the above 

; Further evi.dence. is |>royided by test Vesults obtained in subsequent .years , 
In..botlt 197S-£6 and .1976-77,. r^^lts continue to. show the - superiqrity of high- 
exposure, student? on;^the CareelWest. Even tho,u3*i low-exposure students were C 
presumably feceiving more , career education as Qrojeof acti vi ties >iTicreased over ^ \ 
those yea^s>^the differences^ b|tween.gr(^ups reito in impnifssive. In T|75r76Ul9 -Sf 
70 .'corriparisons (faypr the treatiffent groups at the^ .05 l^el of. signifilance.. In; 
.1976-77^ 17 of (the;^^20 comfJarisoVs favgr 'the treatment group.' -'Twenty-six oV .^lhese 
40^tom]iaxisorfs* oveNt favor- the treatment group.' at the .01 level or 

beyond. ^ "-/^ ^ X ' v ■ ' " ' 

'Evidence that ef/ectWre e'g^etionatly meanfhqTAJl . ThTeveval^tion datav 
developed and presented, in V974^75^ consisted of nOmbers of students tested, mean :< 
^.percentages of items ^corre^-,^-;^- ratios,,.^ sig^nvficance levels,.- At this writing,., 
more detai/ied data are n^lqnger^ available; tharefore, si2e of differences between 
groups ip standard deviation terms canno.t'bfe giv'en. Educational significance is 
better judged instead on o^therJf^ ThesB are: (1) the importance ^of the 

seven* Categories', (2). the modest cost of implementing this program relative to 
the number Qf/^tudents served; and (3) the- ent^ustasffi of prdject and. schojpl staff 
and members pf^ the recipient communities, I=t is noteworth-y that ipJat least -30 
schools, 'sehooUpfficials who w^re reluctant to Tnake' use of ' project/serv.ices 't^^ 
tbecausa.of cor^^^ fofj^ OverbVr.dened teachers and other staff .membWs, iFe^f 
of inop^ased. costs, or-simp^le i^sistance to. ch^ngej were prodded in tb--ig^ by 
parents, teach'^s, and^counselors. Such enthus^'asm "certainly demonstrates the/ 

n the -loGalvrnmmunitv- . ^ ' . i* . --^ 




^prdgram^s ;>er(^ei 




^'mpprtance within the -^ocalvcprranunity 





Table 4 



Mean Percentage of Items Correct on the Careers ^est^, 
/ > ' _t- Rati OS, and Significance Levels ' / 



Elemefgtary/Intermedictte Students (grades 



Educational Awareness 



4-7) 



Jlndwledge af Sk^i 11 S' Required 



/ 



^Knowledge cTf Fac'ior's Contrib-^ 
ting to Job Satisfaction 



o5 /><:?'LCommon Threads .in Jobs 

' Frnnnmir flujA rah occ ' 



Economic Awarelfiess 
iDeci si on* Making V* 
Apprecilrtioris -^and ^Atti tudes 



Knowledge of Career/C|usters Score 



•;' §egon<lary Students (grades ^-12) 
/ , Educational Awareness . 



^ [Knowledge of Skills Required 
*<i |Knowledge of^Fac^jprs Contrib- 
<^ ujnng WJob 'Sati sfa'cti on'^ * 
^^^<^' lCoiTimoh.<iJ^eads in Jobs^ 



§ of 
i tenis 



■ 5" 
5 

. ' 5 - 

- 5 
';9 

4 
7 

7 



H.f^h Exposure Low Exposure 
^ Sample - ^ Sample • 
•■(N 55Q^,. • (N =• 550) 



^ / • 

V 

Significaace 
Level 



51.8 

ei.'i- 
57 .-r 

73.0 
^.9 
68.2- 
65 . 7 
:57.V' 



48.9 
56.6 
53.3 [>' 
^63.5/^ 
54.5 \ 
60.0^ 

58.8 r 



J2,17 
3.05 
' 3.19 
5.88 
"3.51 
5.70 
4.46 



,<TN = .700^- 



■- fi 



ficonomic Awareness 



5> . jf^war^.ness of Career Mobility 



53/7 • 

62.1 

69^'. 3- 

70..&, 

58!6 
45.5 




13/^^6' 



48) 



N A 700J 
42.0 ^ 
50.0. , 



^5.6 
^57.6 



47.1 
35.4 



69 



6.42 

7.08 

6-. 91 

6.34 

5.64 
5 . 11 




59.0 



6 



^•(^ '/^Wenes^ of FactoVs Jnfluenc^ing / 
^<jj^ ij Occupational Structure ' - 

Decision Making . 

- A ppreci a tf ons^ an d'^ifrtt^ 

Knowledge of Calmer Clustm^f^d^^ 
A In ferest in CajWer Clusters, Scored 

\ ' ' , — * r.-^"' '^'T ■ ■ w*>^ v . . . I • . - - . 

^ *propartibn of students indicating "very well-'prepared" for enterin^>/c1u^ter$ ' 
• ^*pjC:Op^rti(^of students indicating 'Nery inter^este^" 



^ ,39.7 
59-.0' 
25.7 



47.4. 
32.7 

47,. 4 

31 .6-. 
24.0 



5.79, 

4.82 > 
8.06 \ 
5 . 92^, 
7 .19- 
3.86 ' 



.<.020 ^ 
<.Opl 
<.001 
<.001 
',<.001 
<.001 
<.001'^ 
<.001 -. 

I ^ 

\."001 

■•<.ooV 

-<'.t)Ol 

<!ooi . 

<.ooA 
<".oo^ 

■<.ooi , 

<.001 

■<^ooi 

.<.-001 

<.ooi . 

<.00-l 



^ t 
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. Evidence that the qfffects are attributab]/e to the intervention , > Tt^ has alreadyr 
lieeo shown that the high and Tow implementaticfh gV6ups were very simi-lar on a number 
of dimensions that -if) diss imilaV might have provided al ternative explanations for 

/ the obtained/ b^een-^grou^ Additiona.l/ evidence that the Pima program 

was responsib;le. foPdijfferencesyin ^student performance lies in the- fact that>all v, 
schools in the high and low Mmplementatiort . groups were^ormally eligible to dFEFin 
project servjjGes. As mentioned'earl.ier,<'this wa§ duMn paf t to" the lack'ofva true, 
"no-exposure" grp^up with Which to- compare (h'iqh-exposurfe stu^^nts ,But it also 
helped ensure tfja^ results were not due toNsystematic' differences in schools who{e . < 

• ^ staff vplunteefed to participate in*the eVfaluation.. / • ' . ' ■ ^ 

' Evidence of general izabil ity to populations^ for tvhibh th^|^ Intervention is intended 
' H^xarvbe confidently stated that this, program is effect;i\(e with a student population, 
made up^of approximately 80% white students and 20% MexicarT-Ameriqan, AmeAjcan Indian, 
> apd' qther minor\fj^ st^nts\ The positive evidence of effecjti veness presented 'for 
1974-75, borne by Vi.milar results, found in. later years but .no t_re ported here, 
Supports the clairfi that 'this program would give excel lent results whfen feplica'teW with' 
- a ^imilair group^of st-uijents. Since-evaluation; results are reported only^for grade Yk.. 



levels 4-T2, the\evidence is rtot general izable j;o^ grades K-3 
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T-UleTX:qMhd Edu.cfttipn Amendments df;i972, PiiBlic Law 92-318 states:? 



• •: "No person ui,\hc United ^taies shall, ^oq 'the basis of sex, be exclucfed - ' ' • 
- from participation fn; .bd^dcni^tl the bertefits of,- or be subjected to ' ' ■ ^ 

, .f.'SGrimrpatmji' Under. any. education ppogram or aGtivityreceivin?- Federal , 
A financial ass st'nnfp" * . , T ^^'"^ i 



f i na nci a 1 ass i s ta n ce." 



. ■-A^'"p^,%^''^°'^»'lthe;Unite^^^^ , s ' 

<:^:%p|f ^n^'^^ ^'^ exc^uded'trom.partiGipation'in, be denied the • ' 

^■ -#VK.P^^^^ or be subjected:;tpdiscriiJiinatio-iunder-^any program or ° ' * ' 

..Section 5Q4;|^^ ' • • ■ ' 

V ' - ;• ■ Any pfcram Or actiViit;^ i^rh'icft re^^s Ft ■d^pal financial' assistance . • 

^vl^may.Woi exclude qualified haifdicapp.ed pe rsons from ' .• 

' ■ ■ . ' " '^f^yf''^^^ must provide equal-bpportu>ay to p-artacipate or benefit; " * ' 

P-^^vide services as ^fective thuge provided to the nonliandicapped: • 
- : .V muy not prpvide diTffr^nt • 

, niec<?ssary to..provide..y2qually effeetive bene/its. . . • 



ISf°;!?' -^^y f?^"<^' ;i°" program or.^ctivity r^deiving- FederS f inanciaLassista'hK" 
or txdrt of a larger entity whiq!-)>eceives Federal financial assistance, ri^ust be •. 
operaterfin compliance with these„^ v . ' ;* ^-v,. , ^' 
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^ tT^f Arl?" - J"vi^^^^ pyplicatioh jvas -^rfepare'^ pursuant to-a- contc'act li^ith the 
_U;S Off ice of lEd-^at^^^^^^ E'dUQk-tion and V,Ware. ■ However, 

points.of view o^,dpjniohS;fe^ '-epresent the policy or - ■ 

•p^sitiorTof thc^Olucex)f- pudation. ' - . u »• • , \ ' ■ ■ 
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